[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 16 Oct 2018 10:12:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. IMPALA-7702: enable fetch incremental stats by default Flips the default from always off to always on for --pull_incremental_statistics. With this setting, the default is for coordinators to fetch incremental stats from catalogd directly (only when computing incremental stats) instead of receiving it from the statestore broadcast. Fetching incremental stats is not applicable when using a CatalogMetaProvider. By making fetch the default, it would require that --pull_incremental_statistics is set to false when enabling CatalogMetaProvider. This change makes --use_local_catalog to take priority over --pull_incremental_statistics so that when both are turned on, only the local catalog setting is enabled. Testing: - manual testing - moved the testing for pull incremental stats out of custom cluster tests since the default flipped - added tests that run with local catalog and pulling incremental stats. Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Reviewed-on: http://gerrit.cloudera.org:8080/11677 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java A tests/custom_cluster/test_incremental_stats.py D tests/custom_cluster/test_pull_stats.py M tests/metadata/test_compute_stats.py 5 files changed, 102 insertions(+), 86 deletions(-) Approvals: Vuk Ercegovac: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 4 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11677/3/tests/custom_cluster/test_incremental_stats.py File tests/custom_cluster/test_incremental_stats.py: http://gerrit.cloudera.org:8080/#/c/11677/3/tests/custom_cluster/test_incremental_stats.py@21 PS3, Line 21: class TestIncrementalStatistics(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 16 Oct 2018 06:46:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1054/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 16 Oct 2018 06:45:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3318/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 16 Oct 2018 06:13:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 3: Code-Review+2 carry +2 -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 16 Oct 2018 06:11:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Hello Bharath Vissapragada, Tianyi Wang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11677 to look at the new patch set (#3). Change subject: IMPALA-7702: enable fetch incremental stats by default .. IMPALA-7702: enable fetch incremental stats by default Flips the default from always off to always on for --pull_incremental_statistics. With this setting, the default is for coordinators to fetch incremental stats from catalogd directly (only when computing incremental stats) instead of receiving it from the statestore broadcast. Fetching incremental stats is not applicable when using a CatalogMetaProvider. By making fetch the default, it would require that --pull_incremental_statistics is set to false when enabling CatalogMetaProvider. This change makes --use_local_catalog to take priority over --pull_incremental_statistics so that when both are turned on, only the local catalog setting is enabled. Testing: - manual testing - moved the testing for pull incremental stats out of custom cluster tests since the default flipped - added tests that run with local catalog and pulling incremental stats. Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 --- M be/src/common/global-flags.cc M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java A tests/custom_cluster/test_incremental_stats.py D tests/custom_cluster/test_pull_stats.py M tests/metadata/test_compute_stats.py 5 files changed, 102 insertions(+), 86 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/11677/3 -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 2: Code-Review+2 (3 comments) carry +2 http://gerrit.cloudera.org:8080/#/c/11677/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11677/2//COMMIT_MSG@18 PS2, Line 18: This patch makes this : easier by not requiring this > rephrase? Done http://gerrit.cloudera.org:8080/#/c/11677/2/be/src/common/global-flags.cc File be/src/common/global-flags.cc: http://gerrit.cloudera.org:8080/#/c/11677/2/be/src/common/global-flags.cc@217 PS2, Line 217: "coordinators."); > Mention that this is not used in local catalog mode? Done http://gerrit.cloudera.org:8080/#/c/11677/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java: http://gerrit.cloudera.org:8080/#/c/11677/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@617 PS2, Line 617: > Add a comment that the pull flag doesn't make much sense in local catalog m Done -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 16 Oct 2018 06:09:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 2: Code-Review+2 (3 comments) http://gerrit.cloudera.org:8080/#/c/11677/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11677/2//COMMIT_MSG@18 PS2, Line 18: This patch makes this : easier by not requiring this rephrase? http://gerrit.cloudera.org:8080/#/c/11677/2/be/src/common/global-flags.cc File be/src/common/global-flags.cc: http://gerrit.cloudera.org:8080/#/c/11677/2/be/src/common/global-flags.cc@217 PS2, Line 217: "coordinators."); Mention that this is not used in local catalog mode? http://gerrit.cloudera.org:8080/#/c/11677/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java: http://gerrit.cloudera.org:8080/#/c/11677/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@617 PS2, Line 617: Add a comment that the pull flag doesn't make much sense in local catalog mode and hence is overridden? -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Mon, 15 Oct 2018 22:08:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 2: Looking at this one. -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Mon, 15 Oct 2018 20:39:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1050/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Oct 2018 20:37:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11677/2/tests/custom_cluster/test_incremental_stats.py File tests/custom_cluster/test_incremental_stats.py: http://gerrit.cloudera.org:8080/#/c/11677/2/tests/custom_cluster/test_incremental_stats.py@21 PS2, Line 21: class TestIncrementalStatistics(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 13 Oct 2018 22:18:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11677 to look at the new patch set (#2). Change subject: IMPALA-7702: enable fetch incremental stats by default .. IMPALA-7702: enable fetch incremental stats by default Flips the default from always off to always on for --pull_incremental_statistics. With this setting, the default is for coordinators to fetch incremental stats from catalogd directly (only when computing incremental stats) instead of receiving it from the statestore broadcast. Fetching incremental stats is not applicable when using a CatalogMetaProvider. By making fetch the default, it would require that --pull_incremental_statistics is set to false when enabling CatalogMetaProvider. This patch makes this easier by not requiring this. Instead, if --use_local_catalog is set to true, that takes priority over --pull_incremental_statistics. Testing: - manual testing - moved the testing for pull incremental stats out of custom cluster tests since the default flipped - added tests that run with local catalog and pulling incremental stats. Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 --- M be/src/common/global-flags.cc M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java A tests/custom_cluster/test_incremental_stats.py D tests/custom_cluster/test_pull_stats.py M tests/metadata/test_compute_stats.py 5 files changed, 97 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/11677/2 -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11677 ) Change subject: IMPALA-7702: enable fetch incremental stats by default .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1048/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 13 Oct 2018 02:06:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7702: enable fetch incremental stats by default
Vuk Ercegovac has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11677 Change subject: IMPALA-7702: enable fetch incremental stats by default .. IMPALA-7702: enable fetch incremental stats by default Flips the default from always off to always on for --pull_incremental_statistics. With this setting, the default is for coordinators to fetch incremental stats from catalogd directly (only when computing incremental stats) instead of receiving it from the statestore broadcast. Fetching incremental stats is not applicable when using a CatalogMetaProvider. By making fetch the default, it would require that --pull_incremental_statistics is set to false when enabling CatalogMetaProvider. This patch makes this easier by not requiring this. Instead, if --use_local_catalog is set to true, that takes priority over --pull_incremental_statistics. Testing: - manual testing Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 --- M be/src/common/global-flags.cc M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java 2 files changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/11677/1 -- To view, visit http://gerrit.cloudera.org:8080/11677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481 Gerrit-Change-Number: 11677 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac
