[Impala-ASF-CR] IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14229 ) Change subject: IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/478/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4fdaf05953373c8d1870b7eface257830c7c6e5 Gerrit-Change-Number: 14229 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 14 Sep 2019 02:41:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14229 ) Change subject: IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/478/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4fdaf05953373c8d1870b7eface257830c7c6e5 Gerrit-Change-Number: 14229 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 14 Sep 2019 02:28:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14229 Change subject: IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger .. IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger Change-Id: Ie4fdaf05953373c8d1870b7eface257830c7c6e5 --- M docs/shared/impala_common.xml M docs/topics/impala_authorization.xml 2 files changed, 33 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/14229/1 -- To view, visit http://gerrit.cloudera.org:8080/14229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie4fdaf05953373c8d1870b7eface257830c7c6e5 Gerrit-Change-Number: 14229 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7975(continued): Improve supportability of the automatic invalidate feature.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14200 ) Change subject: IMPALA-7975(continued): Improve supportability of the automatic invalidate feature. .. IMPALA-7975(continued): Improve supportability of the automatic invalidate feature. This work was done by Sharanitha Harish. This commit adds support to display the metric last-synced-event-id as /metrics#events page whereas previously it was displayed only on the /events page. Testing: Added tests toMetaStoreEventsProcessorTest to verify: 1. If the metric updated as it should when new events were processed. 2. If the metric was not set when the event processor was not active. Change-Id: Iadab4a6cd3ad3c6c1b7931747ac55cd58ce3bb3d Reviewed-on: http://gerrit.cloudera.org:8080/14200 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/util/event-metrics.cc M be/src/util/event-metrics.h M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 6 files changed, 33 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iadab4a6cd3ad3c6c1b7931747ac55cd58ce3bb3d Gerrit-Change-Number: 14200 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7975(continued): Improve supportability of the automatic invalidate feature.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14200 ) Change subject: IMPALA-7975(continued): Improve supportability of the automatic invalidate feature. .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadab4a6cd3ad3c6c1b7931747ac55cd58ce3bb3d Gerrit-Change-Number: 14200 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 14 Sep 2019 02:19:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix DCHECKs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14214 ) Change subject: IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix DCHECKs .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4560/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib96f797bc8a5ba8baf9fb28abd1f645345bbe932 Gerrit-Change-Number: 14214 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Sat, 14 Sep 2019 01:31:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix DCHECKs
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/14214 )
Change subject: IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix
DCHECKs
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14214/1/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/14214/1/be/src/exec/buffered-plan-root-sink.cc@175
PS1, Line 175: while (IsQueueClosedOrEmpty() && sender_state_ ==
SenderState::ROWS_PENDING
: && !state->is_cancelled() && !timed_out) {
> As discussed offline, the contract seems simpler if we make it an invariant
It actually has to check if the query has been cancelled or if the sink has
been closed (because the sink can be closed before is_cancelled() is set to
true). I changed IsQueueClosedOrEmpty() back to IsQueueEmpty(), so now all the
loop have to check if the query has been cancelled or if the sink has been
closed before checking if the queue is empty. Since that check has to be
repeated in multiple places, I moved it into a method called
IsCancelledOrClosed. I'm not sure if thats much better than
IsQueueClosedOrEmpty - the reason I'm adding the helper method is to try and
simply the already complex condition in the while loops.
--
To view, visit http://gerrit.cloudera.org:8080/14214
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96f797bc8a5ba8baf9fb28abd1f645345bbe932
Gerrit-Change-Number: 14214
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Sat, 14 Sep 2019 00:53:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix DCHECKs
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14214 to look at the new patch set (#2). Change subject: IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix DCHECKs .. IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix DCHECKs Adds several "failpoint" tests to test_result_spooling.py. These tests use debug_actions spread throughout buffered-plan-root-sink.cc to trigger failures while result spooling is running. The tests validate that all queries gracefully fail and do not cause any impalad crashes. Fixed a few bugs that came up when adding these tests, as well as the crash reported in IMPALA-8924 (which is now covered by the failpoint tests added in this patch). The first bug fixed was a DCHECK in SpillableRowBatchQueue::IsEmpty() where the method was being called after the queue had been closed. The fix is to only call IsEmpty() if IsOpen() returns true. The second bug was an issue in the cancellation path where BufferedPlanRootSink::GetNext would enter an infinite loop if the query was cancelled and then GetNext was called. The fix is to check the cancellation state in the outer while loop. Testing: * Added new tests to test_result_spooling.py * Ran core tests Change-Id: Ib96f797bc8a5ba8baf9fb28abd1f645345bbe932 --- M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M tests/query_test/test_result_spooling.py 3 files changed, 104 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/14214/2 -- To view, visit http://gerrit.cloudera.org:8080/14214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib96f797bc8a5ba8baf9fb28abd1f645345bbe932 Gerrit-Change-Number: 14214 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4559/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 14 Sep 2019 00:21:46 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Hector Acosta, Philip Zeyliger, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#7). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/flatbuffers/build.sh M source/kudu/build.sh 4 files changed, 121 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/7 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/14180/5/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/14180/5/be/src/service/client-request-state.h@456 PS5, Line 456: Since : /// it does not track rows read from the cache, this counter also tracks the number of : /// materialized rows, so it is used to calculate row_materialization_rate_ > nit: It only counts the number of materialized rows and it is used for deri Done http://gerrit.cloudera.org:8080/#/c/14180/5/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/14180/5/be/src/service/client-request-state.cc@117 PS5, Line 117: num_rows_fetched_cache_counter_ > nit: num_rows_fetched_from_cache_counter_ or num_result_cache_rows_counter_ Done http://gerrit.cloudera.org:8080/#/c/14180/5/tests/hs2/test_fetch_first.py File tests/hs2/test_fetch_first.py: http://gerrit.cloudera.org:8080/#/c/14180/5/tests/hs2/test_fetch_first.py@163 PS5, Line 163: def __get_runtime_profile(self, op_handle): : """Helper method to get the runtime profile from a given operation handle.""" : get_profile_req = ImpalaHiveServer2Service.TGetRuntimeProfileReq() : get_profile_req.operationHandle = op_handle : get_profile_req.sessionHandle = self.session_handle : get_profile_resp = self.hs2_client.GetRuntimeProfile(get_profile_req) : HS2TestSuite.check_response(get_profile_resp) : return get_profile_resp.profile > Feel free to defer the work but we seem to have similar functions elsewhere Yeah, it looks like the hs2_client used in test_observability is a wrapper around impyla (e.g. ImpylaHS2Connection), but the one used here is the Thrift generated ImpalaHiveServer2Service.py. This class probably pre-dated the addition of the impyla wrapper, which accounts for the difference. Agree it would be good to consolidate the two, but will defer it to another JIRA. -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Sep 2019 23:41:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14180 to look at the new patch set (#6). Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. IMPALA-8825: Add additional counters to PlanRootSink Adds the counters RowsSent and RowsSentRate to the PLAN_ROOT_SINK section of the profile: PLAN_ROOT_SINK: - PeakMemoryUsage: 4.01 MB (4202496) - RowBatchGetWaitTime: 0.000ns - RowBatchSendWaitTime: 0.000ns - RowsSent: 10 (10) - RowsSentRate: 416.00 /sec RowsSent tracks the number of rows sent to the PlanRootSink via PlanRootSink::Send. RowsSentRate tracks the rate that rows are sent to the PlanRootSink. Adds the counters NumRowsFetched, NumRowsFetchedFromCache, and RowMaterializationRate to the ImpalaServer section of the profile. ImpalaServer: - ClientFetchWaitTimer: 11.999ms - NumRowsFetched: 10 (10) - NumRowsFetchedFromCache: 10 (10) - RowMaterializationRate: 9.00 /sec - RowMaterializationTimer: 1s007ms NumRowsFetched tracks the total number of rows fetched by the query, but does not include rows fetched from the cache. NumRowsFetchedFromCache tracks the total number of rows fetched from the query results cache. RowMaterializationRate tracks the rate at which rows are materialized. RowMaterializationTimer already existed and tracks how much time is spent materializing rows. Testing: * Added tests to test_fetch_first.py and query_test/test_fetch.py * Enabled some tests in test_fetch_first.py that were pending the completion of IMPALA-8819 * Ran core tests Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M tests/common/impala_test_suite.py M tests/custom_cluster/test_result_spooling.py M tests/hs2/test_fetch_first.py A tests/query_test/test_fetch.py 10 files changed, 199 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/14180/6 -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 5: Code-Review+2 (3 comments) http://gerrit.cloudera.org:8080/#/c/14180/5/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/14180/5/be/src/service/client-request-state.h@456 PS5, Line 456: Since : /// it does not track rows read from the cache, this counter also tracks the number of : /// materialized rows, so it is used to calculate row_materialization_rate_ nit: It only counts the number of materialized rows and it is used for deriving the row_materialization_rate_. http://gerrit.cloudera.org:8080/#/c/14180/5/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/14180/5/be/src/service/client-request-state.cc@117 PS5, Line 117: num_rows_fetched_cache_counter_ nit: num_rows_fetched_from_cache_counter_ or num_result_cache_rows_counter_ or something simpler http://gerrit.cloudera.org:8080/#/c/14180/5/tests/hs2/test_fetch_first.py File tests/hs2/test_fetch_first.py: http://gerrit.cloudera.org:8080/#/c/14180/5/tests/hs2/test_fetch_first.py@163 PS5, Line 163: def __get_runtime_profile(self, op_handle): : """Helper method to get the runtime profile from a given operation handle.""" : get_profile_req = ImpalaHiveServer2Service.TGetRuntimeProfileReq() : get_profile_req.operationHandle = op_handle : get_profile_req.sessionHandle = self.session_handle : get_profile_resp = self.hs2_client.GetRuntimeProfile(get_profile_req) : HS2TestSuite.check_response(get_profile_resp) : return get_profile_resp.profile Feel free to defer the work but we seem to have similar functions elsewhere (https://github.com/apache/impala/blob/master/tests/query_test/test_observability.py#L77) in other tests so it'd be nice to consolidate them in the future. -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Sep 2019 22:29:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7975(continued): Improve supportability of the automatic invalidate feature.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14200 ) Change subject: IMPALA-7975(continued): Improve supportability of the automatic invalidate feature. .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadab4a6cd3ad3c6c1b7931747ac55cd58ce3bb3d Gerrit-Change-Number: 14200 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Sep 2019 22:09:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7975(continued): Improve supportability of the automatic invalidate feature.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14200 ) Change subject: IMPALA-7975(continued): Improve supportability of the automatic invalidate feature. .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4948/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadab4a6cd3ad3c6c1b7931747ac55cd58ce3bb3d Gerrit-Change-Number: 14200 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Sep 2019 22:09:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7975(continued): Improve supportability of the automatic invalidate feature.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14200 ) Change subject: IMPALA-7975(continued): Improve supportability of the automatic invalidate feature. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadab4a6cd3ad3c6c1b7931747ac55cd58ce3bb3d Gerrit-Change-Number: 14200 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Sep 2019 22:08:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4558/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Sep 2019 21:45:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4557/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Sep 2019 21:43:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix DCHECKs
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/14214 )
Change subject: IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix
DCHECKs
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14214/1/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/14214/1/be/src/exec/buffered-plan-root-sink.cc@175
PS1, Line 175: while (IsQueueClosedOrEmpty() && sender_state_ ==
SenderState::ROWS_PENDING
: && !state->is_cancelled() && !timed_out) {
As discussed offline, the contract seems simpler if we make it an invariant
that IsQueueEmpty() cannot be called after the plan root sink has been
cancelled.
IMHO, it seems a bit weird to still loop around if "the queue is closed" part
is true. It seems to be making assumption that state->is_cancelled() is called
afterwards. IMHO, the new interface IsQueueClosedOrEmpty() seems a tad error
prone as it returns true for two possible states and these two states may
potentially lead to drastically different actions (i.e. keep waiting vs
breaking out of the loop).
--
To view, visit http://gerrit.cloudera.org:8080/14214
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96f797bc8a5ba8baf9fb28abd1f645345bbe932
Gerrit-Change-Number: 14214
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Fri, 13 Sep 2019 21:42:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14180 to look at the new patch set (#5). Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. IMPALA-8825: Add additional counters to PlanRootSink Adds the counters RowsSent and RowsSentRate to the PLAN_ROOT_SINK section of the profile: PLAN_ROOT_SINK: - PeakMemoryUsage: 4.01 MB (4202496) - RowBatchGetWaitTime: 0.000ns - RowBatchSendWaitTime: 0.000ns - RowsSent: 10 (10) - RowsSentRate: 416.00 /sec RowsSent tracks the number of rows sent to the PlanRootSink via PlanRootSink::Send. RowsSentRate tracks the rate that rows are sent to the PlanRootSink. Adds the counters NumRowsFetched, NumRowsFetchedFromCache, and RowMaterializationRate to the ImpalaServer section of the profile. ImpalaServer: - ClientFetchWaitTimer: 11.999ms - NumRowsFetched: 10 (10) - NumRowsFetchedFromCache: 10 (10) - RowMaterializationRate: 9.00 /sec - RowMaterializationTimer: 1s007ms NumRowsFetched tracks the total number of rows fetched by the query, but does not include rows fetched from the cache. NumRowsFetchedFromCache tracks the total number of rows fetched from the query results cache. RowMaterializationRate tracks the rate at which rows are materialized. RowMaterializationTimer already existed and tracks how much time is spent materializing rows. Testing: * Added tests to test_fetch_first.py and query_test/test_fetch.py * Enabled some tests in test_fetch_first.py that were pending the completion of IMPALA-8819 * Ran core tests Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M tests/common/impala_test_suite.py M tests/custom_cluster/test_result_spooling.py M tests/hs2/test_fetch_first.py A tests/query_test/test_fetch.py 10 files changed, 199 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/14180/5 -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/service/client-request-state.h@421 PS1, Line 421: /// The number of rows materialized for this query, the counter is not reset if the : /// fetch is restarted. It does not include any rows read from the results cache, since : /// those rows are already materialized. : RuntimeProfile::Counter* rows_materialized_counter_ = nullptr; > It seems a tad confusing to users to interpret these two counters. In essen Done http://gerrit.cloudera.org:8080/#/c/14180/4/tests/hs2/test_fetch_first.py File tests/hs2/test_fetch_first.py: http://gerrit.cloudera.org:8080/#/c/14180/4/tests/hs2/test_fetch_first.py@136 PS4, Line 136: r > flake8: F841 local variable 'num_row_fetched_cache' is assigned to but neve Done http://gerrit.cloudera.org:8080/#/c/14180/4/tests/hs2/test_fetch_first.py@161 PS4, Line 161: n > flake8: F821 undefined name 'num_rows_fetched_cache' Done -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Sep 2019 21:03:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/14180/4/tests/hs2/test_fetch_first.py File tests/hs2/test_fetch_first.py: http://gerrit.cloudera.org:8080/#/c/14180/4/tests/hs2/test_fetch_first.py@136 PS4, Line 136: n flake8: F841 local variable 'num_row_fetched_cache' is assigned to but never used http://gerrit.cloudera.org:8080/#/c/14180/4/tests/hs2/test_fetch_first.py@161 PS4, Line 161: n flake8: F821 undefined name 'num_rows_fetched_cache' -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Sep 2019 21:01:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14180 to look at the new patch set (#4). Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. IMPALA-8825: Add additional counters to PlanRootSink Adds the counters RowsSent and RowsSentRate to the PLAN_ROOT_SINK section of the profile: PLAN_ROOT_SINK: - PeakMemoryUsage: 4.01 MB (4202496) - RowBatchGetWaitTime: 0.000ns - RowBatchSendWaitTime: 0.000ns - RowsSent: 10 (10) - RowsSentRate: 416.00 /sec RowsSent tracks the number of rows sent to the PlanRootSink via PlanRootSink::Send. RowsSentRate tracks the rate that rows are sent to the PlanRootSink. Adds the counters NumRowsFetched, NumRowsFetchedFromCache, and RowMaterializationRate to the ImpalaServer section of the profile. ImpalaServer: - ClientFetchWaitTimer: 11.999ms - NumRowsFetched: 10 (10) - NumRowsFetchedFromCache: 10 (10) - RowMaterializationRate: 9.00 /sec - RowMaterializationTimer: 1s007ms NumRowsFetched tracks the total number of rows fetched by the query, but does not include rows fetched from the cache. NumRowsFetchedFromCache tracks the total number of rows fetched from the query results cache. RowMaterializationRate tracks the rate at which rows are materialized. RowMaterializationTimer already existed and tracks how much time is spent materializing rows. Testing: * Added tests to test_fetch_first.py and query_test/test_fetch.py * Enabled some tests in test_fetch_first.py that were pending the completion of IMPALA-8819 * Ran core tests Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M tests/common/impala_test_suite.py M tests/custom_cluster/test_result_spooling.py M tests/hs2/test_fetch_first.py A tests/query_test/test_fetch.py 10 files changed, 199 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/14180/4 -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8932: addendum - protocol var not defined
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14225 ) Change subject: IMPALA-8932: addendum - protocol var not defined .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75c41a02bc7f1314e48bb5a39b945119264ce478 Gerrit-Change-Number: 14225 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Sep 2019 19:51:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8932: addendum - protocol var not defined
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14225 ) Change subject: IMPALA-8932: addendum - protocol var not defined .. IMPALA-8932: addendum - protocol var not defined Change-Id: I75c41a02bc7f1314e48bb5a39b945119264ce478 Reviewed-on: http://gerrit.cloudera.org:8080/14225 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M shell/impala_shell.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I75c41a02bc7f1314e48bb5a39b945119264ce478 Gerrit-Change-Number: 14225 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/service/client-request-state.h@421 PS1, Line 421: /// The number of rows materialized for this query, the counter is not reset if the : /// fetch is restarted. It does not include any rows read from the results cache, since : /// those rows are already materialized. : RuntimeProfile::Counter* rows_materialized_counter_ = nullptr; > yeah, this is pretty similar to num_rows_fetched_counter_ the difference is It seems a tad confusing to users to interpret these two counters. In essence, these counters only differ if the user somehow restarts the fetch. It may be simpler to just track the number of rows fetched from result cache using a separate counter and have a single counter for number of rows materialized. -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Sep 2019 18:36:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8932: addendum - protocol var not defined
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14225 ) Change subject: IMPALA-8932: addendum - protocol var not defined .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4947/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75c41a02bc7f1314e48bb5a39b945119264ce478 Gerrit-Change-Number: 14225 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Sep 2019 15:31:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8932: addendum - protocol var not defined
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14225 ) Change subject: IMPALA-8932: addendum - protocol var not defined .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75c41a02bc7f1314e48bb5a39b945119264ce478 Gerrit-Change-Number: 14225 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Sep 2019 15:31:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8755: Backend support for Z-ordering
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14080 ) Change subject: IMPALA-8755: Backend support for Z-ordering .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4556/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0200748ce3e65ebc5d3530f794c0f80aa335a2ab Gerrit-Change-Number: 14080 Gerrit-PatchSet: 8 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 13 Sep 2019 13:39:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8755: Backend support for Z-ordering
Norbert Luksa has posted comments on this change. (
http://gerrit.cloudera.org:8080/14080 )
Change subject: IMPALA-8755: Backend support for Z-ordering
..
Patch Set 8:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/14080/7/be/src/util/tuple-row-compare-test.cc
File be/src/util/tuple-row-compare-test.cc:
http://gerrit.cloudera.org:8080/#/c/14080/7/be/src/util/tuple-row-compare-test.cc@94
PS7, Line 94: Tuple* tuple_mem = Tuple::Create(sizeof(char) +
GetSize(args...), _perm_pool_);
> line too long (92 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/14080/5/be/src/util/tuple-row-compare.cc
File be/src/util/tuple-row-compare.cc:
http://gerrit.cloudera.org:8080/#/c/14080/5/be/src/util/tuple-row-compare.cc@314
PS5, Line 314:
> nit: Can we come up with a better name? Maybe GetZDimensionValue() or somet
Done
http://gerrit.cloudera.org:8080/#/c/14080/5/be/src/util/tuple-row-compare.cc@334
PS5, Line 334: turn Comp
> Maybe you could add a DCHECK(false); as well, and maybe a TODO comment. If
Done
http://gerrit.cloudera.org:8080/#/c/14080/5/be/src/util/tuple-row-compare.cc@383
PS5, Line 383: rn
> nit: since you use 'lhs' and 'rhs' at other places, maybe rename 'v1' and '
Done
http://gerrit.cloudera.org:8080/#/c/14080/7/be/src/util/tuple-row-compare.cc
File be/src/util/tuple-row-compare.cc:
http://gerrit.cloudera.org:8080/#/c/14080/7/be/src/util/tuple-row-compare.cc@209
PS7, Line 209: Status TupleRowLexicalComparator::CodegenCompare(LlvmCodeGen*
codegen,
> line too long (93 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/14080/7/be/src/util/tuple-row-compare.cc@323
PS7, Line 323: constexpr uint64_t mask64 = 0x8000;
> line too long (95 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/14080/7/be/src/util/tuple-row-compare.cc@395
PS7, Line 395: case TYPE_TIMESTAMP: {
> line too long (91 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/14080
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0200748ce3e65ebc5d3530f794c0f80aa335a2ab
Gerrit-Change-Number: 14080
Gerrit-PatchSet: 8
Gerrit-Owner: Norbert Luksa
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Norbert Luksa
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 13 Sep 2019 12:58:41 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8755: Backend support for Z-ordering
Norbert Luksa has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/14080 ) Change subject: IMPALA-8755: Backend support for Z-ordering .. IMPALA-8755: Backend support for Z-ordering This change depends on gerrit.cloudera.org/#/c/13955/ (Frontend support for Z-ordering) The commit adds a Comperator based on Z-ordering. See in detail: https://en.wikipedia.org/wiki/Z-order_curve The comperator instead of calculating the Z-values of the rows, looks for the column with the most significant dimension, and compares the values of this column only. The most significant dimension will be the one where the compared values have the highest different bits. The algorithm requires values of the same binary representation, but this can be relaxed. Currently, strings, varchars, floats and doubles are not supported. Testing: * Added unit tests. * Currently, some tests are missing. * Run manual tests, comparing 4-column values with 4-bit integers, for all possible combinations. Checked the result by calculating the Z-value for each comparison. * Tested performance on various data, getting great results. Change-Id: I0200748ce3e65ebc5d3530f794c0f80aa335a2ab --- M be/src/exec/exchange-node.cc M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partial-sort-node.h M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/exec/topn-node.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/sorter.cc M be/src/runtime/sorter.h M be/src/util/CMakeLists.txt A be/src/util/tuple-row-compare-test.cc M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h 16 files changed, 776 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/14080/8 -- To view, visit http://gerrit.cloudera.org:8080/14080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0200748ce3e65ebc5d3530f794c0f80aa335a2ab Gerrit-Change-Number: 14080 Gerrit-PatchSet: 8 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
