[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 15 Apr 2021 09:44:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This change disables the overlap min/max filter test for hdfs in erasure coding, due to the query plan change (from 3-node scan to 2-node scan) which splits the row groups among scan nodes differently. The SkipIfEC class in test harness skip.py is enhanced with a new skip reason 'different_scan_split' to facilitate this action. Testing: 1. Ran unit tests; 2. Ran core tests. Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Reviewed-on: http://gerrit.cloudera.org:8080/17289 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/common/skip.py M tests/query_test/test_runtime_filters.py 2 files changed, 4 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 7 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 15 Apr 2021 04:02:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7077/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 15 Apr 2021 04:02:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 5: retest -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 15 Apr 2021 02:33:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7070/ -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 14 Apr 2021 17:46:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7070/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 14 Apr 2021 11:57:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7069/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 13 Apr 2021 23:37:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 13 Apr 2021 23:37:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 13 Apr 2021 23:36:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 13 Apr 2021 17:28:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8564/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 13 Apr 2021 15:57:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This change disables the overlap min/max filter test for hdfs in erasure coding, due to the query plan change (from 3-node scan to 2-node scan) which splits the row groups among scan nodes differently. The SkipIfEC class in test harness skip.py is enhanced with a new skip reason 'different_scan_split' to facilitate this action. Testing: 1. Ran unit tests; 2. Ran core tests. Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f --- M tests/common/skip.py M tests/query_test/test_runtime_filters.py 2 files changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/17289/4 -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG@7 PS2, Line 7: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky nit: maybe this is more accurate: Skip TestOverlapMinMaxFilters.test_overlap_min_max_filters when EC is enabled -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 13 Apr 2021 06:36:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG@10 PS2, Line 10: due to the query plan change (from 3-node scan to : 2-node scan) which splits the row groups among scan nodes differently. > One more piece of details on instance 2. The overlap ratio of 0 indicates t Ah, I see! Thanks for the detailed explanation! It's really helpful. http://gerrit.cloudera.org:8080/#/c/17289/2/tests/query_test/test_runtime_filters.py File tests/query_test/test_runtime_filters.py: http://gerrit.cloudera.org:8080/#/c/17289/2/tests/query_test/test_runtime_filters.py@270 PS2, Line 270: @SkipIfNotHdfsMinicluster.scheduling IIUC, the issue is due to the underlying parquet files have different stats between EC and non-EC runs. It's not about scheduling. So I suggest we add a new skip type in SkipIfEC and use that instead, e.g. SkipIfEC.different_stats, and adding this in SkipIfEC (tests/common/skip.py) different_stats = pytest.mark.skipif(IS_EC, reason="Parquet/ORC files that are generated" " in tests have different stats in EC mode due to larger block group size.") -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 13 Apr 2021 02:15:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG@10 PS2, Line 10: due to the query plan change (from 3-node scan to : 2-node scan) which splits the row groups among scan nodes differently. > Here are some details if it is not too much :-) The bottom line is that onl One more piece of details on instance 2. The overlap ratio of 0 indicates that the entire row group is filtered out (which is very good). We do not drill down to figure out how many pages there are in the group, for the sake of reporting. -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 12 Apr 2021 20:49:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG@10 PS2, Line 10: due to the query plan change (from 3-node scan to : 2-node scan) which splits the row groups among scan nodes differently. > It's not clear to me why no pages are filtered by the min-max filter. Could Here are some details if it is not too much :-) The bottom line is that only 2 nodes are involved and the split is such that in instance 1, the overlap ratio (0.749748) is higher than the threshold (0.5) which implies no min/max filtering at the page and row level. In instance 2, the overlap ration is 0 which implies no min/max filtering. Combined, no page level filtering is done and thus the total pages filtered is 0. The details is purposely left out in the commit message. Executor instance 1: 1732 E RowGroup Debug: Try to filter out a rowgroup via overlap predicate: MonotonicMillis=13987739, fid=1, SchemaNode=optional int64 l_orderkey [i:0 d:1 r:0], columnType=BIGINT, overlap ratio=0.749748, threshold=0.5, worthiness=0, enabled for page=0, enabled for row=0, data min=1, data max=3209607, content=BigIntMinMaxFilter(min=224167, max=2630562, always_false=false), always_true=false) 1778 E - NumRuntimeFilteredPages: 0 (0) 1779 E - NumRuntimeFilteredRowGroups: 0 (0) Executor instance 2 1919 E RowGroup Debug: Try to filter out a rowgroup via overlap predicate: MonotonicMillis=13987773, fid=1, SchemaNode=optional int64 l_orderkey [i:0 d:1 r:0], columnType=BIGINT, overlap ratio=0, threshold=0.5, worthiness=1, enabled for page=1, enabled for row=0, data min=3209632, data max=600, content=BigIntMinMaxFilter(min=224167, max=2630562, always_false=false), always_true=false) 1965 E - NumRuntimeFilteredPages: 0 (0) 1966 E - NumRuntimeFilteredRowGroups: 1 (1) -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 12 Apr 2021 17:45:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17289/2//COMMIT_MSG@10 PS2, Line 10: due to the query plan change (from 3-node scan to : 2-node scan) which splits the row groups among scan nodes differently. It's not clear to me why no pages are filtered by the min-max filter. Could you help to add more details? BTW, I left a comment in the JIRA about how to reproduce this issue locally: https://issues.apache.org/jira/browse/IMPALA-10532?focusedCommentId=17319265=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17319265 -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 12 Apr 2021 10:09:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8538/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 10 Apr 2021 01:15:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17289 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 10 Apr 2021 01:13:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17289 Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This change disables the overlap min/max filter test for hdfs in erasure coding, due to the query plan change (from 3-node scan to 2-node scan) which splits the row groups among scan nodes differently. Testing: 1. Ran unit tests; 2. Ran core tests. Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f --- M tests/query_test/test_runtime_filters.py 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/17289/2 -- To view, visit http://gerrit.cloudera.org:8080/17289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I527de530f7db1ce959e7ef2ae3ced18677221c9f Gerrit-Change-Number: 17289 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17252 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 10: (4 comments) http://gerrit.cloudera.org:8080/#/c/17252/10//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17252/10//COMMIT_MSG@7 PS10, Line 7: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky > In my opinion, we should split this into 2 separate changes. Agree to separate two changes. http://gerrit.cloudera.org:8080/#/c/17252/10/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/17252/10/be/src/runtime/coordinator.cc@1586 PS10, Line 1586: params.min_max_filter().always_false() Add a DCHECK to make sure always_false() return false. http://gerrit.cloudera.org:8080/#/c/17252/10/be/src/runtime/coordinator.cc@1637 PS10, Line 1637: unnecessary blank line http://gerrit.cloudera.org:8080/#/c/17252/10/be/src/runtime/coordinator.cc@1639 PS10, Line 1639: this Could we use filter-id, instead of this pointer? -- To view, visit http://gerrit.cloudera.org:8080/17252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326317833979efcbe02ce6c95ad80133dd5c7964 Gerrit-Change-Number: 17252 Gerrit-PatchSet: 10 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 08 Apr 2021 17:51:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/17252 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 10: (3 comments) http://gerrit.cloudera.org:8080/#/c/17252/10//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17252/10//COMMIT_MSG@7 PS10, Line 7: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky In my opinion, we should split this into 2 separate changes. One is focus on fixing the test_overlap_min_max_filters, and the other is for the query profile improvement. http://gerrit.cloudera.org:8080/#/c/17252/10/be/src/exec/scan-node.cc File be/src/exec/scan-node.cc: http://gerrit.cloudera.org:8080/#/c/17252/10/be/src/exec/scan-node.cc@236 PS10, Line 236: Current " :"time(ms): $2", I think this is redundant and potentially confuse users who try to debug query profile. The information from wait_time (Maximum arrival delay) should be sufficient. As far as I know, MonotonicMilis mostly used to calculate time interval, not to represent system time. Besides, filter arrival time is already displayed under host profile like this: impala-executor-000-1:27000: Filter 16 arrival: 662ms Filter 14 arrival: 10s454ms Filter 6 arrival: 10s514ms Filter 8 arrival: 5m46s Filter 9 arrival: 5m46s Filter 10 arrival: 5m46s Filter 0 arrival: 7m29s Filter 1 arrival: 7m29s Filter 2 arrival: 7m29s http://gerrit.cloudera.org:8080/#/c/17252/10/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/17252/10/be/src/runtime/coordinator.cc@1614 PS10, Line 1614: minmaxDisabled_ = true; I think we should maintain the same behavior as bloom_filter_ in here, making the logic consistent for both filter type. I understand that this change is made to distinguish between 1) an actual AlwaysTrue filter that originate from executor, vs 2) AlwaysTrue filter that comes from disabling by coordinator. Instead, what if we have a new boolean variable to track 1). Say, a boolean variable 'always_true_minmax_received_'. Coordinator::FilterDebugString() then should refer this boolean variable instead of checking minmax_filterPB.always_true(). Similarly also for AlwaysFalse filter arrival. -- To view, visit http://gerrit.cloudera.org:8080/17252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326317833979efcbe02ce6c95ad80133dd5c7964 Gerrit-Change-Number: 17252 Gerrit-PatchSet: 10 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 08 Apr 2021 16:25:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17252 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8514/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326317833979efcbe02ce6c95ad80133dd5c7964 Gerrit-Change-Number: 17252 Gerrit-PatchSet: 10 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Apr 2021 16:27:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/17252 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This change disables the overlap min/max filter test for hdfs in erasure coding, due to the query plan change (from 3-node scan to 2-node scan) which splits the row groups among scan nodes differently. The change also improves how a coordinator behaves when a just arriving min/max filter is the last one to arrive or is always true. Previously, the coordinator disables the corresponding filter representation by setting it to Always True, which makes it impossible to differentiate a true AlwaysTrue filter (say, set in the hash join building step) from the one being disabled. A dedicated Boolean variable minmaxDisabled_ is introduced to record the disabled state. The Always True state of a filter is never altered. The enhancement improves the display of the min and max column in "Filter routing table" and "Final filter table" in profile. These two columns now display the following possible values. 1. 'PartialUpdates' - The min and the max are partially updated; 2. 'AlwaysTrue' - The filter is always true; 3. 'AlwaysFalse'- The filter is always false; 4. Real values - The filter is neither always true or false, fully updated with the min/max real values. A third change introduced is to record, in profile for scan node, the arrival time of min/max filters (in elapsed time since the system is rebooted obtained by calling MonotonicMillis()). It can help the diagnosis of late arrival of filters, when compared with the elpased time when a row group is filtered with these filters. Testing: 1. Ran unit tests; 2. Ran core tests. Change-Id: I326317833979efcbe02ce6c95ad80133dd5c7964 --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/scan-node.cc M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/util/min-max-filter.cc M be/src/util/min-max-filter.h M tests/query_test/test_runtime_filters.py 7 files changed, 112 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/17252/10 -- To view, visit http://gerrit.cloudera.org:8080/17252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I326317833979efcbe02ce6c95ad80133dd5c7964 Gerrit-Change-Number: 17252 Gerrit-PatchSet: 10 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17252 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 9: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/8513/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326317833979efcbe02ce6c95ad80133dd5c7964 Gerrit-Change-Number: 17252 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Apr 2021 15:33:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532: TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/17252 ) Change subject: IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532: TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This change disables the overlap min/max filter test for hdfs in erasure coding, due to the query plan change (from 3-node scan to 2-node scan) which splits the row groups among scan nodes differently. The change also improves how a coordinator behaves when a just arriving min/max filter is the last one to arrive or is always true. Previously, the coordinator disables the corresponding filter representation by setting it to Always True, which makes it impossible to differentiate a true AlwaysTrue filter (say, set in the hash join building step) from the one being disabled. A dedicated Boolean variable minmaxDisabled_ is introduced to record the disabled state. The Always True state of a filter is never altered. The enhancement improves the display of the min and max column in "Filter routing table" and "Final filter table" in profile. These two columns now display the following possible values. 1. 'PartialUpdates' - The min and the max are partially updated; 2. 'AlwaysTrue' - The filter is always true; 3. 'AlwaysFalse'- The filter is always false; 4. Real values - The filter is neither always true or false, fully updated with the min/max real values. A third change introduced is to record, in profile for scan node, the arrival time of min/max filters (in elapsed time since the system is rebooted obtained by calling MonotonicMillis()). It can help the diagnosis of late arrival of filters, when compared with the elpased time when a row group is filtered with these filters. Testing: 1. Ran unit tests; 2. Ran core tests. Change-Id: I326317833979efcbe02ce6c95ad80133dd5c7964 --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/scan-node.cc M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/util/min-max-filter.cc M be/src/util/min-max-filter.h M tests/query_test/test_runtime_filters.py 7 files changed, 112 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/17252/9 -- To view, visit http://gerrit.cloudera.org:8080/17252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I326317833979efcbe02ce6c95ad80133dd5c7964 Gerrit-Change-Number: 17252 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This patch addresses the flakiness seen with a particular test within overlap_min_max_filters by allowing the sum of NumRuntimeFilteredPages to be greater than an expected value. Previously, such a sum can only be equal to the expected value and is not sufficient for various test conditions in which the scan of the parquet data files can start before the arrival of a runtime filter. The extension in test_result_verifier.py allows '>' and '<' condition to be expressed for aggregation(SUM, ), such as aggregation(SUM, NumRuntimeFilteredPages)> 80. Testing: - Ran TestOverlapMinMaxFilters. Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Reviewed-on: http://gerrit.cloudera.org:8080/17111 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test M tests/common/test_result_verifier.py 2 files changed, 52 insertions(+), 13 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 7 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 26 Feb 2021 22:52:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 26 Feb 2021 17:07:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6923/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 26 Feb 2021 17:07:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/17111/5/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test File testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test: http://gerrit.cloudera.org:8080/#/c/17111/5/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test@9 PS5, Line 9: CREATE TABLE tpch_parquet.lineitem_orderkey_only(l_orderkey bigint) > I see there are already other instances like this in this file, so I guess Sounds like a good idea. Will make the change in my other commit in this area. Will leave it as is for this commit. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 26 Feb 2021 01:03:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 5: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/17111/5/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test File testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test: http://gerrit.cloudera.org:8080/#/c/17111/5/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test@9 PS5, Line 9: CREATE TABLE tpch_parquet.lineitem_orderkey_only(l_orderkey bigint) I see there are already other instances like this in this file, so I guess its fine to leave as is, but I wanted to note that I think a better way of doing this rather than creating the table in the tpch_parquet database (which potentially leaves it in a weird state if one of the test cases before the "drop table" below fails), would be to use the 'unique_database' functionality and create the table there, which would guarantee that its always dropped even if an error is encountered. -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 25 Feb 2021 21:21:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8238/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 25 Feb 2021 14:36:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This patch addresses the flakiness seen with a particular test within overlap_min_max_filters by allowing the sum of NumRuntimeFilteredPages to be greater than an expected value. Previously, such a sum can only be equal to the expected value and is not sufficient for various test conditions in which the scan of the parquet data files can start before the arrival of a runtime filter. The extension in test_result_verifier.py allows '>' and '<' condition to be expressed for aggregation(SUM, ), such as aggregation(SUM, NumRuntimeFilteredPages)> 80. Testing: - Ran TestOverlapMinMaxFilters. Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 --- M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test M tests/common/test_result_verifier.py 2 files changed, 52 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/17111/5 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 25 Feb 2021 00:16:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test File testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test: http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test@34 PS4, Line 34: 80 > Should we skip this test for sanitizer builds, which are slow? The failures were with exhaustive release build. Tested Sanitize build on my R box and observed the same result. http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test@34 PS4, Line 34: 80 > Well, the scanner should wait at least RUNTIME_FILTER_WAIT_TIME_MS before s Yes, I checked and think you are right. The wait time used during all failed tests was set at 6ms and the actual arrival time was around 370ms. In all failed tests (exhaustive release), the total number of filtered pages (NumRuntimeFilteredPages) is 128. Therefore the use of 80 as the lower bound is pretty reasonable. http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test@34 PS4, Line 34: 80 > How was this lower bound determined? What happens if things are really slow It was based on the error report seen -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 24 Feb 2021 05:34:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test File testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test: http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test@34 PS4, Line 34: 80 > Well, the scanner should wait at least RUNTIME_FILTER_WAIT_TIME_MS before s Should we skip this test for sanitizer builds, which are slow? -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Feb 2021 22:01:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test File testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test: http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test@34 PS4, Line 34: 80 > It was based on several error reports where the total number of pages filte Well, the scanner should wait at least RUNTIME_FILTER_WAIT_TIME_MS before starting, so assuming that's set high enough to ensure they arrive before it starts (which is what we've always relied on in other similar runtime filter tests) I think the total sum would be deterministic. Relying on that still leaves the possibility that the test is flaky in unusually slow environments, of course, but it has the nice property that when the test does run correctly we've ensured that the filter is applied correctly (eg. by making this a lower bound instead of a precise count there's a certain class of bugs where the filters are being misapplied and not filtering things that should be filtered that the test would not catch anymore). So, it might be nice to just increase RUNTIME_FILTER_WAIT_TIME_MS instead of what you've done here. On the other hand, we've already increased that value a few times in the past due to this sort of flakiness, and we can't just keep increasing it forever. I'll leave it up to you. -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 23 Feb 2021 20:51:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test File testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test: http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test@34 PS4, Line 34: 80 > How was this lower bound determined? What happens if things are really slow It was based on several error reports where the total number of pages filtered is in the high 80s (https://master-02.jenkins.cloudera.com/job/impala-asf-master-exhaustive-release/1049/testReport/junit/query_test.test_runtime_filters/TestOverlapMinMaxFilters/test_overlap_min_max_filters_protocol__beeswax___exec_optionbatch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___True___abort_on_error___1___exec_single_node_rows_threshold___0table_format__parquet_none_/). In my R box, the total is over 90. The total number of pages in the two tables remains the same, across these two testing environment. Since the scanner starts the process without waiting for the arrival of filters, this un-determinedness is always there. Setting a value at 80 provides a pretty reasonable lower bound, IMHO :-). -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 23 Feb 2021 20:34:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test File testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test: http://gerrit.cloudera.org:8080/#/c/17111/4/testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test@34 PS4, Line 34: 80 How was this lower bound determined? What happens if things are really slow, could even more of the scan happen before the filter arrives, making this test still flaky? I guess in the extreme case this number could even end up as 0, if the entire scan happens before the filter arrives? -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 23 Feb 2021 20:12:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8209/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Feb 2021 20:09:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This patch addresses the flakiness seen with a particular test within overlap_min_max_filters by allowing the sum of NumRuntimeFilteredPages to be greater than an expected value. Previously, such a sum can only be equal to the expected value and is not sufficient for various test conditions in which the scan of the parquet data files can start before the arrival of a runtime filter. The extension in test_result_verifier.py allows '>' and '<' condition to be expressed for aggregation(SUM, ), such as aggregation(SUM, NumRuntimeFilteredPages)> 80. Testing: - Ran TestOverlapMinMaxFilters. Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 --- M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test M tests/common/test_result_verifier.py 2 files changed, 32 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/17111/4 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8207/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Feb 2021 19:55:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8206/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Feb 2021 19:51:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8205/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 1 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Feb 2021 19:47:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17111/3/tests/common/test_result_verifier.py File tests/common/test_result_verifier.py: http://gerrit.cloudera.org:8080/#/c/17111/3/tests/common/test_result_verifier.py@541 PS3, Line 541: r flake8: E126 continuation line over-indented for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Feb 2021 19:36:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This patch addresses the flakiness seen with a particular test within overlap_min_max_filters by allowing the sum of NumRuntimeFilteredPages to be greater than an expected value. Previously, such a sum can only be equal to the expected value and is not sufficient for various test conditions in which the scan of the parquet data files can start before the arrival of a runtime filter. The extension in test_result_verifier.py allows '>' and '<' condition to be expressed for aggregation(SUM, ), such as aggregation(SUM, NumRuntimeFilteredPages)> 80. Testing: - Ran TestOverlapMinMaxFilters. Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 --- M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test M tests/common/test_result_verifier.py 2 files changed, 32 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/17111/3 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17111/2/tests/common/test_result_verifier.py File tests/common/test_result_verifier.py: http://gerrit.cloudera.org:8080/#/c/17111/2/tests/common/test_result_verifier.py@540 PS2, Line 540: 0 flake8: E501 line too long (92 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Feb 2021 19:25:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This patch addresses the flakiness seen with a particular test within overlap_min_max_filters by allowing the sum of NumRuntimeFilteredPages to be greater than an expected value. Previously, such a sum can only be equal to the expected value and is not sufficient for various test conditions in which the scan of the parquet data files can start before the arrival of a runtime filter. The extension in test_result_verifier.py allows '>' and '<' condition to be expressed for aggregation(SUM, ), such as aggregation(SUM, NumRuntimeFilteredPages)> 80. Testing: - Ran TestOverlapMinMaxFilters. Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 --- M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test M tests/common/test_result_verifier.py 2 files changed, 31 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/17111/2 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17111 ) Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/17111/1/tests/common/test_result_verifier.py File tests/common/test_result_verifier.py: http://gerrit.cloudera.org:8080/#/c/17111/1/tests/common/test_result_verifier.py@540 PS1, Line 540: 0 flake8: E501 line too long (92 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 1 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Feb 2021 19:20:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10532 TestOverlapMinMaxFilters.test overlap min max filters seems flaky
Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17111 Change subject: IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky .. IMPALA-10532 TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky This patch addresses the flakyness seen with a particular test within overlap_min_max_filters by allowing the sum of NumRuntimeFilteredPages to be greater than a value. Previously, such a sum can only be equal to an expected value and is not sufficient for various test conditions in which the scan of the parquet data files can start before the arrival of a runtime filter. The extension to test_result_verifier.py allows '>' and '<' condition to be expressed for aggregation(SUM, ), such as aggregation(SUM, NumRuntimeFilteredPages)> 80. Testing: - Ran TestOverlapMinMaxFilters. Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 --- M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test M tests/common/test_result_verifier.py 2 files changed, 31 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/17111/1 -- To view, visit http://gerrit.cloudera.org:8080/17111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I93940a104bfb2d68cb1d41d7e303348190fd5972 Gerrit-Change-Number: 17111 Gerrit-PatchSet: 1 Gerrit-Owner: Qifan Chen
