[Impala-ASF-CR] IMPALA-7466: Improve readability of describe authorization tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11278 ) Change subject: IMPALA-7466: Improve readability of describe authorization tests .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6244b6dbafbd3827d488588a16e66dbf15d0e115 Gerrit-Change-Number: 11278 Gerrit-PatchSet: 4 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 Aug 2018 05:56:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7466: Improve readability of describe authorization tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11278 ) Change subject: IMPALA-7466: Improve readability of describe authorization tests .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3065/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6244b6dbafbd3827d488588a16e66dbf15d0e115 Gerrit-Change-Number: 11278 Gerrit-PatchSet: 4 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 Aug 2018 05:56:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7466: Improve readability of describe authorization tests
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11278 ) Change subject: IMPALA-7466: Improve readability of describe authorization tests .. Patch Set 3: > Patch Set 3: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3053/ Got a timeout while running the GVO. Re-running the GVO. -- To view, visit http://gerrit.cloudera.org:8080/11278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6244b6dbafbd3827d488588a16e66dbf15d0e115 Gerrit-Change-Number: 11278 Gerrit-PatchSet: 3 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 Aug 2018 05:56:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7457. statestore: allow filtering by key prefix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11253 ) Change subject: IMPALA-7457. statestore: allow filtering by key prefix .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11253 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ddcf3bfaf16bc3cd1ba01100e948ff142a67620 Gerrit-Change-Number: 11253 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 22 Aug 2018 05:19:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11280 ) Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3064/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 Gerrit-Change-Number: 11280 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 22 Aug 2018 05:08:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7447. Evict LocalCatalog cache entries based on size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11231 ) Change subject: IMPALA-7447. Evict LocalCatalog cache entries based on size .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia96af49b35c17e505b7b6785e78d140939085d91 Gerrit-Change-Number: 11231 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 22 Aug 2018 05:08:05 + Gerrit-HasComments: No
[Impala-ASF-CR] cleanup: remove RuntimeState::exec env
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11269 ) Change subject: cleanup: remove RuntimeState::exec_env_ .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3063/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4a1b1bdd41e3d10982b3a4bdb0217e716b4df67f Gerrit-Change-Number: 11269 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 22 Aug 2018 03:03:06 + Gerrit-HasComments: No
[Impala-ASF-CR] cleanup: remove RuntimeState::exec env
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11269 ) Change subject: cleanup: remove RuntimeState::exec_env_ .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4a1b1bdd41e3d10982b3a4bdb0217e716b4df67f Gerrit-Change-Number: 11269 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 22 Aug 2018 03:03:05 + Gerrit-HasComments: No
[Impala-ASF-CR] cleanup: remove RuntimeState::exec env
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11269 ) Change subject: cleanup: remove RuntimeState::exec_env_ .. Patch Set 4: Hit IMPALA-7418 -- To view, visit http://gerrit.cloudera.org:8080/11269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4a1b1bdd41e3d10982b3a4bdb0217e716b4df67f Gerrit-Change-Number: 11269 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 22 Aug 2018 03:02:37 + Gerrit-HasComments: No
[Impala-ASF-CR] cleanup: remove RuntimeState::exec env
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11269 ) Change subject: cleanup: remove RuntimeState::exec_env_ .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3060/ -- To view, visit http://gerrit.cloudera.org:8080/11269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4a1b1bdd41e3d10982b3a4bdb0217e716b4df67f Gerrit-Change-Number: 11269 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 22 Aug 2018 03:01:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7402: fix DCHECK when releasing reservation in scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11283 ) Change subject: IMPALA-7402: fix DCHECK when releasing reservation in scan .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Gerrit-Change-Number: 11283 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 22 Aug 2018 02:22:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7402: fix DCHECK when releasing reservation in scan
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11283 ) Change subject: IMPALA-7402: fix DCHECK when releasing reservation in scan .. IMPALA-7402: fix DCHECK when releasing reservation in scan The bug is that ScannerContext::Stream::GetNextBuffer(), when reading past the end of a scan range and ScanRange::GetNext() returned cancelled, did not wait for buffers owned by the scan range to be freed. Subsequent code assumes that all buffers allocated by the scanner are freed after HdfsScanner::Close() returns, but this was not guaranteed. The fix is to strengthen the post-condition of ScanRange::GetNext() so that buffers are guaranteed to be returned when GetNext() returns CANCELLED. Testing: Added a unit test that tests the new invariant. Manually tested that this fixed the regression by inserting a 10ms sleep in BufferPool::FreeBuffer() and looping the test that failed. Ran DiskIoMgrStressTest overnight and ran core tests. Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Reviewed-on: http://gerrit.cloudera.org:8080/11283 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc 3 files changed, 62 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Gerrit-Change-Number: 11283 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11280 ) Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/446/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 Gerrit-Change-Number: 11280 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 22 Aug 2018 02:20:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7447. Evict LocalCatalog cache entries based on size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11231 ) Change subject: IMPALA-7447. Evict LocalCatalog cache entries based on size .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/445/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia96af49b35c17e505b7b6785e78d140939085d91 Gerrit-Change-Number: 11231 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 22 Aug 2018 02:20:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7213, IMPALA-7241: Port ReportExecStatus() RPC to use KRPC
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10855 ) Change subject: IMPALA-7213, IMPALA-7241: Port ReportExecStatus() RPC to use KRPC .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.cc File be/src/runtime/coordinator-backend-state.cc: http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.cc@265 PS10, Line 265: DCHECK(!instance_stats->done_ || : report_version == instance_stats->last_report_version_); > isn't it possible (though unlikely) that you have some old report sitting i Yup. That's actually a DCHECK which Sailesh ran into when testing IMPALA-4063. Removed in patch 11. http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/query-state.cc File be/src/runtime/query-state.cc: http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/query-state.cc@287 PS10, Line 287: ReportVersion > if I understand the threading correctly, there are two entry points for sen To answer the question about your second race, I think it's impossible today because we explicitly turn off the reporting thread before sending the final profile (https://github.com/apache/impala/blob/master/be/src/runtime/fragment-instance-state.cc#L479-L496) The first race is also kind of impossible today because we wait for the report thread to exit before sending the final report but yes, it's possible that the RPC in the report thread fails somehow and we just drop those error statuses on the floor. -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 Aug 2018 01:58:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7447. Evict LocalCatalog cache entries based on size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11231 ) Change subject: IMPALA-7447. Evict LocalCatalog cache entries based on size .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3061/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia96af49b35c17e505b7b6785e78d140939085d91 Gerrit-Change-Number: 11231 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 22 Aug 2018 01:47:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7457. statestore: allow filtering by key prefix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11253 ) Change subject: IMPALA-7457. statestore: allow filtering by key prefix .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3062/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11253 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ddcf3bfaf16bc3cd1ba01100e948ff142a67620 Gerrit-Change-Number: 11253 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 22 Aug 2018 01:47:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/11280 )
Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic
updates
..
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/11280/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/11280/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@275
PS2, Line 275: LOG.trace("Request for DB metadata for {}: {}", dbName,
hit.getRef() ? "hit":"miss");
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11280/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@346
PS2, Line 346: dbName, tableName, resp.table_info.hms_table,
resp.object_version_number);
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/11280/2/tests/metadata/test_ddl.py
File tests/metadata/test_ddl.py:
http://gerrit.cloudera.org:8080/#/c/11280/2/tests/metadata/test_ddl.py@572
PS2, Line 572: )
flake8: E501 line too long (91 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/11280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984
Gerrit-Change-Number: 11280
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 22 Aug 2018 01:41:32 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7447. Evict LocalCatalog cache entries based on size
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/11231 ) Change subject: IMPALA-7447. Evict LocalCatalog cache entries based on size .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/11231/3/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java File fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java: http://gerrit.cloudera.org:8080/#/c/11231/3/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java@111 PS3, Line 111: partialRefs); > its primarily to make sure that the flag we pass in for expiration is hooke k, I added a TODO at the cache instantiation site to add e2e tests to stress eviction -- To view, visit http://gerrit.cloudera.org:8080/11231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia96af49b35c17e505b7b6785e78d140939085d91 Gerrit-Change-Number: 11231 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 22 Aug 2018 01:38:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Hello Bharath Vissapragada, Tianyi Wang, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11280 to look at the new patch set (#2). Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates .. IMPALA-7469. Invalidate LocalCatalog cache based on topic updates This implements cache invalidation inside CatalogdMetaProvider. The design is as follows: - when the catalogd collects updates into the statestore topic, it now adds an additional entry for each table and database. These additional entries are minimal - they only include the object's name, but no metadata. - the old-style topic entries are prefixed with a '1:' whereas the new minimal entries are prefixed with a '2:'. The impalad will subscribe to one or the other prefix depending on whether it is running with --use_local_catalog. Thus, old impalads will not be confused by the new entries and vice versa. - when the impalad gets these topic updates, it forwards them through to the catalog implementation. The LocalCatalog implementation forwards them to the CatalogdMetaProvider, which uses them to invalidate cached metadata as appropriate. This patch includes some basic unit tests. I also did some manual testing by connecting to different impalads and verifying that a session connected to impalad #1 saw the effects of DDLs made by impalad #2 within a short period of time (the statestore topic update frequency). Existing end-to-end tests cover these code paths pretty thoroughly: - if we didn't automatically invalidate the cache on a coordinator in response to DDL operations, then any test which expects to "read its own writes" (eg access a table after creating one) would fail - if we didn't propagate invalidations via the statestore, then all of the tests that use sync_ddl would fail. I verified the test coverage above using some of the tests in test_ddl.py -- I selectively commented out a few of the invalidation code paths in the new code and verified that tests failed until I re-introduced them. Along the way I also improved test_ddl so that, when this code is broken, it properly fails with a timeout. It also has a bit of expanded coverage for both the SYNC_DDL and non-SYNC cases. One notable exception here is the implementation of SYNC_DDL for INVALIDATE METADATA. This turned out to be complex to implement, so I left a lengthy TODO describing the issue. Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 --- M be/src/service/impala-server.cc M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/common/Pair.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M tests/metadata/test_ddl.py 14 files changed, 525 insertions(+), 88 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/11280/2 -- To view, visit http://gerrit.cloudera.org:8080/11280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 Gerrit-Change-Number: 11280 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7447. Evict LocalCatalog cache entries based on size
Hello Bharath Vissapragada, Tianyi Wang, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11231 to look at the new patch set (#6). Change subject: IMPALA-7447. Evict LocalCatalog cache entries based on size .. IMPALA-7447. Evict LocalCatalog cache entries based on size This pulls in the 'sizeof' library from ehcache (Apache-licensed) and uses it to implement size-based eviction of cache entries in LocalCatalog. This is difficult to test without being quite fragile to small changes in the cached structures. However, I added a simple unit test as a general sanity-check that it is computing some reasonable result. Change-Id: Ia96af49b35c17e505b7b6785e78d140939085d91 --- M be/src/runtime/exec-env.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java 7 files changed, 113 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11231/6 -- To view, visit http://gerrit.cloudera.org:8080/11231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia96af49b35c17e505b7b6785e78d140939085d91 Gerrit-Change-Number: 11231 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7466: Improve readability of describe authorization tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11278 ) Change subject: IMPALA-7466: Improve readability of describe authorization tests .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3053/ -- To view, visit http://gerrit.cloudera.org:8080/11278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6244b6dbafbd3827d488588a16e66dbf15d0e115 Gerrit-Change-Number: 11278 Gerrit-PatchSet: 3 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 Aug 2018 00:58:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7213, IMPALA-7241: Port ReportExecStatus() RPC to use KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10855 ) Change subject: IMPALA-7213, IMPALA-7241: Port ReportExecStatus() RPC to use KRPC .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/444/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 Aug 2018 00:44:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting mem limit
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting mem_limit .. Patch Set 7: (5 comments) http://gerrit.cloudera.org:8080/#/c/11157/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11157/7//COMMIT_MSG@10 PS7, Line 10: With this patch the mem_limit of a query is automatically set based on : the mem_limit set in the query options and the mem_estimate calculated : by the planner. > could you go into more detail what this means? how is the "output" mem_limi For the new behavior (Assuming either min or max memlimit is set in the pool): if mem_limit is set in query options: use that and if 'strict-min-max-query-mem-limit' is true, enforce the min/max limits defined in the pool config. else: mem_limit = max(mem_estiamte, min_mem_limit_required_to_accomodate_largest_min_reservation) and enforce min/max limits defined in the pool config on this value. http://gerrit.cloudera.org:8080/#/c/11157/7//COMMIT_MSG@16 PS7, Line 16: If these are set to zero > is it if any or all are set to zero? Both need to be set to zero http://gerrit.cloudera.org:8080/#/c/11157/7//COMMIT_MSG@18 PS7, Line 18: if false, the mem_limit defined in : the query options is used directly and the max/min limits are not : enforced on it. > does that mean this option effectively disables this new behavior entirely? this only disables a part of it for the case when mem_limit is defined in the query options. This was added as a workaround for the case where a query cannot run if the min/max limits are applied to it http://gerrit.cloudera.org:8080/#/c/11157/7/common/thrift/ImpalaInternalService.thrift File common/thrift/ImpalaInternalService.thrift: http://gerrit.cloudera.org:8080/#/c/11157/7/common/thrift/ImpalaInternalService.thrift@783 PS7, Line 783: // If TRUE, the mem_limit query option will not be bounded by the max/min query mem : // limits specified for the pool. Default is FALSE. need to update this as with the change in name it would mean the opposite http://gerrit.cloudera.org:8080/#/c/11157/7/common/thrift/metrics.json File common/thrift/metrics.json: http://gerrit.cloudera.org:8080/#/c/11157/7/common/thrift/metrics.json@53 PS7, Line 53: "description": "Resource Pool $0 Max Query Memory Limit", > did you sanity check that all these metrics look right on the metrics page? Yes, I manually checked them. also one of them, namely admission-controller.pool-max-query-mem-limit is used in one of the tests that dynamically changes the pool config -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 22 Aug 2018 00:28:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7213, IMPALA-7241: Port ReportExecStatus() RPC to use KRPC
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/10855 )
Change subject: IMPALA-7213, IMPALA-7241: Port ReportExecStatus() RPC to use
KRPC
..
Patch Set 11:
(15 comments)
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/exec/hdfs-parquet-table-writer.h
File be/src/exec/hdfs-parquet-table-writer.h:
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/exec/hdfs-parquet-table-writer.h@199
PS10, Line 199: ParquetInsertStatsPB parquet_insert_stats_;
nit: should #include the appropriate .pb.h here ("include-what-you-use")
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.h
File be/src/runtime/coordinator-backend-state.h:
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.h@60
PS10, Line 60: Coordinator* coord
I think this can probably change back to being const if you take the suggestion
below. Also please add a doc comment indicating that the Coordinator must
remain valid until the BackendState object is destructed.
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.h@176
PS10, Line 176: last_report_ti
nit: I think the term "sequence number" is more usual here -- "version" to me
sounds like each update is always a "replace the previous update", whereas
"sequence" indicates that there should not be gaps
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.h@220
PS10, Line 220: /// Reference to the owning coordinator's 'dml_exec_state_'.
This "back pointer" still seems error-prone to me. I think the object lifetimes
and relationships will be clearer if you pass this in as an out-argument to
every call to ApplyExecStatusReport(), so that the struct becomes single-owner
without "escaping" pointers.
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.cc@265
PS10, Line 265: }
:
isn't it possible (though unlikely) that you have some old report sitting in
the service queue long enough that you could get some late delivery with a
sequence number that is < last_report_version, not just equal to?
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.cc@267
PS10, Line 267: inline bool Coordinator::BackendState::IsDone() const {
I think a VLOG_QUERY about the skipped RPC is probably useful
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/coordinator-backend-state.cc@294
PS10, Line 294: instance_stats->Update(
nit: why not:
const Status& instance_status = instance_exec_status.status();
it's a bit more conventional for making a copy (if necessary) and will also
just take a reference if safe to do so.
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/query-state.cc@287
PS10, Line 287:
if I understand the threading correctly, there are two entry points for sending
a report: one is periodic during the query, and the other is on query
completion ("done"). Once the "done" report is sent, any other reports will be
ignored.
Does this mean that it's possible we can race as follows?
- periodic report constructs a request including calling GetUnreportedErrors
- perhaps gets a SerivceUnavailable and goes to sleep
- the query finishes and triggers a 'done' report, with no errors (because they
got consumed by the above already)
- the first report retries and its report gets ignored because the fragment is
already "done"
thus we'd be losing the reported errors. Perhaps this is a known/existing bug,
but maybe its' worth documenting the behavior in a TODO here if not a JIRA?
On a related note, I'm afraid the following race might cause a problem:
- exec thread reporting "done" allocates seqno 10
- makes a request, gets "unavailable" or whatever (or just goes to sleep for a
few millis)
- profile reporting thread allocates seqno 11 and sends a request, which is
accepted by the server
-- the server side increases the sequence number to 11
- the exec thread sends the "I'm done" message with seqno 10
- the server now ignores the "done" request and the query hangs
Does that seem like a possibility to you? Maybe we could trigger such a race in
a test by setting the service pool size for this endpoint to e smaller than the
number of fragments and increasing the report frequency to once every
millisecond or something? Given that the effect of a bug in this area is
serious (hung queries) we should be pretty careful about testing I think.
http://gerrit.cloudera.org:8080/#/c/10855/10/be/src/runtime/query-state.cc@375
PS10, Line 375:
MonoDelta::FromMilliseconds(FLAGS_backend_client_rpc_timeout_ms));
[Impala-ASF-CR] IMPALA-7213, IMPALA-7241: Port ReportExecStatus() RPC to use KRPC
Hello Sailesh Mukil, Todd Lipcon, Impala Public Jenkins, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10855 to look at the new patch set (#11). Change subject: IMPALA-7213, IMPALA-7241: Port ReportExecStatus() RPC to use KRPC .. IMPALA-7213, IMPALA-7241: Port ReportExecStatus() RPC to use KRPC This change converts ReportExecStatus() RPC from thrift based RPC to KRPC. This is done in part of the preparation for fixing IMPALA-2990 as we can take advantage of TCP connection multiplexing in KRPC to avoid overwhelming the coordinator with too many connections by reducing the number of TCP connection to one for each executor. This patch also introduces a new service pool for all query execution control related RPCs in the future so that control commands from coordinators aren't blocked by long-running DataStream services' RPCs. The majority of this patch is mechanical conversion of some Thrift structures used in ReportExecStatus() RPC to Protobuf. Note that the runtime profile is still retained as a Thrift structure as Impala clients will still fetch query profiles using Thrift RPCs. This also avoids duplicating the serialization implementation in both Thrift and Protobuf for the runtime profile. The Thrift runtime profiles are serialized and sent as a sidecar in ReportExecStatus() RPC. This patch also fixes IMPALA-7241 which may lead to duplicated dml stats being applied. The fix is by adding a monotonically increasing version number for fragment instances' reports. The coordinator will ignore any report smaller than or equal to the version in the last report. Testing done: core build. Added some targeted test cases for profile serialization failure and RPC retry. Change-Id: I7638583b433dcac066b87198e448743d90415ebe --- M be/src/benchmarks/expr-benchmark.cc M be/src/catalog/catalog-util.cc M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/hbase-table-sink.cc M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-parquet-table-writer.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-writer.cc M be/src/exec/hdfs-table-writer.h M be/src/rpc/jni-thrift-util.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.h M be/src/runtime/backend-client.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/dml-exec-state.cc M be/src/runtime/dml-exec-state.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/test-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/scheduler-test-util.cc M be/src/service/CMakeLists.txt M be/src/service/client-request-state.cc M be/src/service/client-request-state.h A be/src/service/control-service.cc A be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h A be/src/util/error-util-internal.h M be/src/util/error-util-test.cc M be/src/util/error-util.cc M be/src/util/error-util.h M be/src/util/runtime-profile.cc M be/src/util/uid-util.h M bin/impala-config.sh M common/protobuf/CMakeLists.txt M common/protobuf/common.proto A common/protobuf/control_service.proto M common/protobuf/data_stream_service.proto M common/protobuf/row_batch.proto M common/protobuf/rpc_test.proto M common/thrift/ImpalaInternalService.thrift M tests/custom_cluster/test_rpc_exception.py M tests/failure/test_failpoints.py 60 files changed, 1,106 insertions(+), 670 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/10855/11 -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] cleanup: remove RuntimeState::exec env
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11269 ) Change subject: cleanup: remove RuntimeState::exec_env_ .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4a1b1bdd41e3d10982b3a4bdb0217e716b4df67f Gerrit-Change-Number: 11269 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:39:23 + Gerrit-HasComments: No
[Impala-ASF-CR] cleanup: remove RuntimeState::exec env
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11269 ) Change subject: cleanup: remove RuntimeState::exec_env_ .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3060/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4a1b1bdd41e3d10982b3a4bdb0217e716b4df67f Gerrit-Change-Number: 11269 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:39:24 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-6772: add ORC 1.5.2
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11254 ) Change subject: IMPALA-6772: add ORC 1.5.2 .. Patch Set 2: Build ID is 191-78f370624a -- To view, visit http://gerrit.cloudera.org:8080/11254 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24430321d0712bc3eda83d77a1a577270eb1e78f Gerrit-Change-Number: 11254 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:37:01 + Gerrit-HasComments: No
[Impala-ASF-CR] cleanup: remove RuntimeState::exec env
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/11269 ) Change subject: cleanup: remove RuntimeState::exec_env_ .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4a1b1bdd41e3d10982b3a4bdb0217e716b4df67f Gerrit-Change-Number: 11269 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:36:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/10847/10/be/src/util/minidump.cc File be/src/util/minidump.cc: http://gerrit.cloudera.org:8080/#/c/10847/10/be/src/util/minidump.cc@119 PS10, Line 119: sigaction(SIGUSR1, _action, NULL); > While you're here, maybe check the return value to make sure this was succe oldact->sa_handler has a value 1 so can't have a check for nullptr -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 10 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 21 Aug 2018 23:31:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7399: Emit a junit xml report when trapping errors
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11257 ) Change subject: IMPALA-7399: Emit a junit xml report when trapping errors .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/443/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc Gerrit-Change-Number: 11257 Gerrit-PatchSet: 8 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 21 Aug 2018 23:30:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 10: (2 comments) > (4 comments) http://gerrit.cloudera.org:8080/#/c/10847/10/be/src/common/init.cc File be/src/common/init.cc: http://gerrit.cloudera.org:8080/#/c/10847/10/be/src/common/init.cc@179 PS10, Line 179: : > nit: space on wrong side of : Done http://gerrit.cloudera.org:8080/#/c/10847/10/be/src/common/init.cc@297 PS10, Line 297: sigaction(SIGTERM, , nullptr); > We should at least check the return value here. We also should pass an empt There is a value 1 displayed for old_action.sa_handler so can't have a nullptr check that will hit the assert -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 10 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 21 Aug 2018 23:30:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting mem limit
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting mem_limit .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/11157/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11157/7//COMMIT_MSG@10 PS7, Line 10: With this patch the mem_limit of a query is automatically set based on : the mem_limit set in the query options and the mem_estimate calculated : by the planner. could you go into more detail what this means? how is the "output" mem_limit computed from the query option "mem_limit"? And does that mean the query option mem_limit is no longer a limit on memory consumption? http://gerrit.cloudera.org:8080/#/c/11157/7//COMMIT_MSG@16 PS7, Line 16: If these are set to zero is it if any or all are set to zero? http://gerrit.cloudera.org:8080/#/c/11157/7//COMMIT_MSG@18 PS7, Line 18: if false, the mem_limit defined in : the query options is used directly and the max/min limits are not : enforced on it. does that mean this option effectively disables this new behavior entirely? or does it disable only parts of it? -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:29:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting mem limit
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting mem_limit .. Patch Set 7: (3 comments) I think I'm basically ok with this modulo the remaining issues with naming and the question about code coverage. I think I'll still want to do another pass over the logic later to convince myself that it's correct. http://gerrit.cloudera.org:8080/#/c/11157/7/be/src/runtime/query-exec-mgr.cc File be/src/runtime/query-exec-mgr.cc: http://gerrit.cloudera.org:8080/#/c/11157/7/be/src/runtime/query-exec-mgr.cc@76 PS7, Line 76: const missed some unnecessary consts here and below. http://gerrit.cloudera.org:8080/#/c/11157/7/be/src/runtime/query-state.cc File be/src/runtime/query-state.cc: http://gerrit.cloudera.org:8080/#/c/11157/7/be/src/runtime/query-state.cc@58 PS7, Line 58: const unnecessary const http://gerrit.cloudera.org:8080/#/c/11157/7/common/thrift/metrics.json File common/thrift/metrics.json: http://gerrit.cloudera.org:8080/#/c/11157/7/common/thrift/metrics.json@53 PS7, Line 53: "description": "Resource Pool $0 Max Query Memory Limit", did you sanity check that all these metrics look right on the metrics page? -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:19:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7399: Emit a junit xml report when trapping errors
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11257 ) Change subject: IMPALA-7399: Emit a junit xml report when trapping errors .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/442/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc Gerrit-Change-Number: 11257 Gerrit-PatchSet: 7 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 21 Aug 2018 23:14:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7412: width bucket() function overflows too easily
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11282 ) Change subject: IMPALA-7412: width_bucket() function overflows too easily .. Patch Set 1: Code-Review+2 Might be good to run the fuzz test overnight before merging if you haven't already done that. -- To view, visit http://gerrit.cloudera.org:8080/11282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I68262698144029ef7f54e027e586eaf105f36ab3 Gerrit-Change-Number: 11282 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Tue, 21 Aug 2018 23:13:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7399: Emit a junit xml report when trapping errors
Hello Lars Volker, Michael Brown, Nithya Janarthanan, Philip Zeyliger, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11257 to look at the new patch set (#9). Change subject: IMPALA-7399: Emit a junit xml report when trapping errors .. IMPALA-7399: Emit a junit xml report when trapping errors This patch will cause a junitxml file to be emitted in the case of errors in build scripts. Instead of simply echoing a message to the console, we set up a trap function that also writes out the the junit xml report that can be consumed by jenkins.impala.io. Main things to pay attention to: - New file that gets sourced by all bash scripts when trapping within bash scripts: https://gerrit.cloudera.org/c/11257/1/bin/report_build_error.sh - Installation of the python lib into impala-python venv for use from within python files: https://gerrit.cloudera.org/c/11257/1/bin/impala-python-common.sh - Change to the generate_junitxml.py file itself, for ease of https://gerrit.cloudera.org/c/11257/1/lib/python/impala_py_lib/jenkins/generate_junitxml.py Most of the other changes are to source the new report_build_error.sh script to set up the trap function. Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc --- M bin/clean-cmake.sh M bin/clean.sh M bin/create-test-configuration.sh M bin/create_testdata.sh M bin/distcc/distcc_server_setup.sh M bin/impala-python-common.sh M bin/jenkins/all-tests.sh M bin/jenkins/build-all-flag-combinations.sh M bin/jenkins/build-only.sh M bin/make_impala.sh A bin/report_build_error.sh M bin/run-all-tests.sh M bin/run-backend-tests.sh M bin/start-catalogd.sh M bin/start-impalad.sh M bin/start-statestored.sh M buildall.sh M infra/python/bootstrap_virtualenv.py M lib/python/impala_py_lib/jenkins/generate_junitxml.py M shell/make_shell_tarball.sh M testdata/bin/check-schema-diff.sh M testdata/bin/compute-table-stats.sh M testdata/bin/copy-data-sources.sh M testdata/bin/copy-udfs-udas.sh M testdata/bin/create-load-data.sh M testdata/bin/create-table-many-blocks.sh M testdata/bin/generate-load-nested.sh M testdata/bin/kill-all.sh M testdata/bin/kill-hbase.sh M testdata/bin/kill-hive-server.sh M testdata/bin/kill-java-service.sh M testdata/bin/kill-sentry-service.sh M testdata/bin/load-hive-builtins.sh M testdata/bin/load-metastore-snapshot.sh M testdata/bin/load-test-warehouse-snapshot.sh M testdata/bin/run-all.sh M testdata/bin/run-hbase.sh M testdata/bin/run-hive-server.sh M testdata/bin/run-mini-dfs.sh M testdata/bin/run-sentry-service.sh M testdata/bin/setup-hdfs-env.sh M testdata/cluster/admin M testdata/datasets/tpcds/preload M testdata/datasets/tpch/preload M tests/run-custom-cluster-tests.sh M tests/run-process-failure-tests.sh 46 files changed, 159 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11257/9 -- To view, visit http://gerrit.cloudera.org:8080/11257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc Gerrit-Change-Number: 11257 Gerrit-PatchSet: 9 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7402: fix DCHECK when releasing reservation in scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11283 ) Change subject: IMPALA-7402: fix DCHECK when releasing reservation in scan .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Gerrit-Change-Number: 11283 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:09:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7402: fix DCHECK when releasing reservation in scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11283 ) Change subject: IMPALA-7402: fix DCHECK when releasing reservation in scan .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3059/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Gerrit-Change-Number: 11283 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:09:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7399: Sample usage of junit xml reporting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11257 ) Change subject: IMPALA-7399: Sample usage of junit xml reporting .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/441/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc Gerrit-Change-Number: 11257 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 21 Aug 2018 23:08:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7402: fix DCHECK when releasing reservation in scan
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11283 ) Change subject: IMPALA-7402: fix DCHECK when releasing reservation in scan .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Gerrit-Change-Number: 11283 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:06:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting mem limit
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting mem_limit .. Patch Set 7: @Dan Any thoughts on the newly added pool attributes? -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 23:06:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7399: Sample usage of junit xml reporting
Hello Lars Volker, Michael Brown, Nithya Janarthanan, Philip Zeyliger, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11257 to look at the new patch set (#7). Change subject: IMPALA-7399: Sample usage of junit xml reporting .. IMPALA-7399: Sample usage of junit xml reporting This draft is intended to show sample usage of the junit reporting script. I'm posting as a draft to get general impressions. If this seems like roughly the right idea, I'll clean it up and post a real review. Things to pay attention to: - New file that gets sourced by all bash scripts when trapping within bash scripts: https://gerrit.cloudera.org/c/11257/1/bin/report_build_error.sh - Installation of the python lib into impala-python venv for use from within python files: https://gerrit.cloudera.org/c/11257/1/bin/impala-python-common.sh - Change to the generate_junitxml.py file itself, for ease of use from python. (Honestly, up until now, I had been thinking that it would ONLY be used as a command line tool, until I realized that we probably want to import it inside the python files too.) https://gerrit.cloudera.org/c/11257/1/lib/python/impala_py_lib/jenkins/generate_junitxml.py - Sample bash usage: e.g.: https://gerrit.cloudera.org/c/11257/1/testdata/bin/run-all.sh - Sample python usage: https://gerrit.cloudera.org/c/11257/1/bin/bootstrap_toolchain.py - Artificial build error: https://gerrit.cloudera.org/c/11257/1/bin/start-statestored.sh Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc --- M bin/clean-cmake.sh M bin/clean.sh M bin/create-test-configuration.sh M bin/create_testdata.sh M bin/distcc/distcc_server_setup.sh M bin/impala-python-common.sh M bin/jenkins/all-tests.sh M bin/jenkins/build-all-flag-combinations.sh M bin/jenkins/build-only.sh M bin/make_impala.sh A bin/report_build_error.sh M bin/run-all-tests.sh M bin/run-backend-tests.sh M bin/start-catalogd.sh M bin/start-impalad.sh M bin/start-statestored.sh M buildall.sh M infra/python/bootstrap_virtualenv.py M lib/python/impala_py_lib/jenkins/generate_junitxml.py M shell/make_shell_tarball.sh M testdata/bin/check-schema-diff.sh M testdata/bin/compute-table-stats.sh M testdata/bin/copy-data-sources.sh M testdata/bin/copy-udfs-udas.sh M testdata/bin/create-load-data.sh M testdata/bin/create-table-many-blocks.sh M testdata/bin/generate-load-nested.sh M testdata/bin/kill-all.sh M testdata/bin/kill-hbase.sh M testdata/bin/kill-hive-server.sh M testdata/bin/kill-java-service.sh M testdata/bin/kill-sentry-service.sh M testdata/bin/load-hive-builtins.sh M testdata/bin/load-metastore-snapshot.sh M testdata/bin/load-test-warehouse-snapshot.sh M testdata/bin/run-all.sh M testdata/bin/run-hbase.sh M testdata/bin/run-hive-server.sh M testdata/bin/run-mini-dfs.sh M testdata/bin/run-sentry-service.sh M testdata/bin/setup-hdfs-env.sh M testdata/cluster/admin M testdata/datasets/tpcds/preload M testdata/datasets/tpch/preload M tests/run-custom-cluster-tests.sh M tests/run-process-failure-tests.sh 46 files changed, 158 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11257/7 -- To view, visit http://gerrit.cloudera.org:8080/11257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc Gerrit-Change-Number: 11257 Gerrit-PatchSet: 7 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7399: Sample usage of junit xml reporting
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/11257 )
Change subject: IMPALA-7399: Sample usage of junit xml reporting
..
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11257/6/bin/report_build_error.sh
File bin/report_build_error.sh:
http://gerrit.cloudera.org:8080/#/c/11257/6/bin/report_build_error.sh@24
PS6, Line 24: $IMPALA_HOME/bin/generate_junitxml.py --step "${FILENAME%.*}"
--error "Error in $0 at line $1: $ERROR_MSG"
line too long (108 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/11257
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc
Gerrit-Change-Number: 11257
Gerrit-PatchSet: 6
Gerrit-Owner: David Knupp
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Brown
Gerrit-Reviewer: Nithya Janarthanan
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Tue, 21 Aug 2018 22:36:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7399: Sample usage of junit xml reporting
Hello Lars Volker, Michael Brown, Nithya Janarthanan, Philip Zeyliger, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11257 to look at the new patch set (#6). Change subject: IMPALA-7399: Sample usage of junit xml reporting .. IMPALA-7399: Sample usage of junit xml reporting This draft is intended to show sample usage of the junit reporting script. I'm posting as a draft to get general impressions. If this seems like roughly the right idea, I'll clean it up and post a real review. Things to pay attention to: - New file that gets sourced by all bash scripts when trapping within bash scripts: https://gerrit.cloudera.org/c/11257/1/bin/report_build_error.sh - Installation of the python lib into impala-python venv for use from within python files: https://gerrit.cloudera.org/c/11257/1/bin/impala-python-common.sh - Change to the generate_junitxml.py file itself, for ease of use from python. (Honestly, up until now, I had been thinking that it would ONLY be used as a command line tool, until I realized that we probably want to import it inside the python files too.) https://gerrit.cloudera.org/c/11257/1/lib/python/impala_py_lib/jenkins/generate_junitxml.py - Sample bash usage: e.g.: https://gerrit.cloudera.org/c/11257/1/testdata/bin/run-all.sh - Sample python usage: https://gerrit.cloudera.org/c/11257/1/bin/bootstrap_toolchain.py - Artificial build error: https://gerrit.cloudera.org/c/11257/1/bin/start-statestored.sh Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc --- M bin/clean-cmake.sh M bin/clean.sh M bin/create-test-configuration.sh M bin/create_testdata.sh M bin/distcc/distcc_server_setup.sh M bin/impala-python-common.sh M bin/jenkins/all-tests.sh M bin/jenkins/build-all-flag-combinations.sh M bin/jenkins/build-only.sh M bin/make_impala.sh A bin/report_build_error.sh M bin/run-all-tests.sh M bin/run-backend-tests.sh M bin/start-catalogd.sh M bin/start-impalad.sh M bin/start-statestored.sh M buildall.sh M infra/python/bootstrap_virtualenv.py M lib/python/impala_py_lib/jenkins/generate_junitxml.py M shell/make_shell_tarball.sh M testdata/bin/check-schema-diff.sh M testdata/bin/compute-table-stats.sh M testdata/bin/copy-data-sources.sh M testdata/bin/copy-udfs-udas.sh M testdata/bin/create-load-data.sh M testdata/bin/create-table-many-blocks.sh M testdata/bin/generate-load-nested.sh M testdata/bin/kill-all.sh M testdata/bin/kill-hbase.sh M testdata/bin/kill-hive-server.sh M testdata/bin/kill-java-service.sh M testdata/bin/kill-sentry-service.sh M testdata/bin/load-hive-builtins.sh M testdata/bin/load-metastore-snapshot.sh M testdata/bin/load-test-warehouse-snapshot.sh M testdata/bin/run-all.sh M testdata/bin/run-hbase.sh M testdata/bin/run-hive-server.sh M testdata/bin/run-mini-dfs.sh M testdata/bin/run-sentry-service.sh M testdata/bin/setup-hdfs-env.sh M testdata/cluster/admin M testdata/datasets/tpcds/preload M testdata/datasets/tpch/preload M tests/run-custom-cluster-tests.sh M tests/run-process-failure-tests.sh 46 files changed, 158 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11257/6 -- To view, visit http://gerrit.cloudera.org:8080/11257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idd62045bb43357abc2b89a78afff499149d3c3fc Gerrit-Change-Number: 11257 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. IMPALA-6844: Fix possible NULL dereference in to_date() builtin If result.ptr allocation fails for some reason inside the StringVal constructor, we still overwrite result.len and continue. This change checks that the StringVal pointer is not NULL before dereferencing it, and returns NULL if it is. Testing: Added a test case of the to_date() function to alloc-fail-init.test to leverage the fault injector --stress_fn_ctx_alloc. Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Reviewed-on: http://gerrit.cloudera.org:8080/11286 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M be/src/exprs/timestamp-functions-ir.cc M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-init.test 2 files changed, 7 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 4 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 3 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 22:06:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 3 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 22:02:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7433: reduce logging on executors
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11202 ) Change subject: IMPALA-7433: reduce logging on executors .. Patch Set 4: (1 comment) I think there are good ideas but I was hoping to keep this change limited to modifying log messages rather than collecting new information to log. http://gerrit.cloudera.org:8080/#/c/11202/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11202/4//COMMIT_MSG@19 PS4, Line 19: I0813 12:10:50.374167 31259 query-state.cc:485] Instance completed. instance_id=fd4ae28bc993236e:27343be10003 #in-flight=1 status=OK > At this point the interesting question is often "Which one is the last inst Can't we infer this from an "Executing instance" log without a corresponding "Instance completed" log. It might be useful to occasionally log the set of executing fragments plus additional info for long-running queries, e.g. in the ReportExecStatus() thread. That would probably be more useful than logging it opportunistically when a different fragment finishes. -- To view, visit http://gerrit.cloudera.org:8080/11202 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c1db44acc6def2b05a4fd032c63716e08cdf5ff Gerrit-Change-Number: 11202 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 21:58:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7465: fix test kudu scan mem usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11285 ) Change subject: IMPALA-7465: fix test_kudu_scan_mem_usage .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca Gerrit-Change-Number: 11285 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 21:38:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7449: Fix network throughput calculation of DataStreamSender
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11241 ) Change subject: IMPALA-7449: Fix network throughput calculation of DataStreamSender .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11241 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I344ac76c0a1a49b4da3d37d2c547f3d5051ebe24 Gerrit-Change-Number: 11241 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 21 Aug 2018 21:29:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7449: Fix network throughput calculation of DataStreamSender
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11241 ) Change subject: IMPALA-7449: Fix network throughput calculation of DataStreamSender .. IMPALA-7449: Fix network throughput calculation of DataStreamSender Currently, the network throughput presented in the query profile for DataStreamSender is computed by dividing the total bytes sent by the total network time which is the sum of observed network time of all individual RPCs. This is wrong in general and may only make sense if the network throughput is fixed. In addition, RPCs are asynchronous and they overlap with each other. So, dividing the total byte sent by network throughput may result in time which exceeds the wall clock time, making it impossible to interpret. This change fixes the problem by measuring the network throughput of each individual RPC and uses a summary counter to track avg/min/max of network throughputs instead. Change-Id: I344ac76c0a1a49b4da3d37d2c547f3d5051ebe24 Reviewed-on: http://gerrit.cloudera.org:8080/11241 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h 2 files changed, 14 insertions(+), 14 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11241 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I344ac76c0a1a49b4da3d37d2c547f3d5051ebe24 Gerrit-Change-Number: 11241 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] [DRAFT] IMPALA-6189: Add thread watchdogs for HDFS IO calls
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10696 ) Change subject: [DRAFT] IMPALA-6189: Add thread watchdogs for HDFS IO calls .. Patch Set 4: I was looking into other stuff and this got sidetracked. Will address the comments and upload a new PS soon. I agree this will be very useful in RCA'ing hangs. -- To view, visit http://gerrit.cloudera.org:8080/10696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28e918761c120043d332b034450208eaf34e3e2b Gerrit-Change-Number: 10696 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 21 Aug 2018 21:22:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7447. Evict LocalCatalog cache entries based on size
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11231 ) Change subject: IMPALA-7447. Evict LocalCatalog cache entries based on size .. Patch Set 5: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/11231/3/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java File fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java: http://gerrit.cloudera.org:8080/#/c/11231/3/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java@111 PS3, Line 111: partialRefs); > There is a way to fake timing but I didn't want to duplicate the testing of its primarily to make sure that the flag we pass in for expiration is hooked up as expected. I'm fine with an end-to-end custom-cluster test covering that. perhaps add a todo for that? -- To view, visit http://gerrit.cloudera.org:8080/11231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia96af49b35c17e505b7b6785e78d140939085d91 Gerrit-Change-Number: 11231 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 21 Aug 2018 21:13:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7470: SentryServicePinger logs error messages on success
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11281 ) Change subject: IMPALA-7470: SentryServicePinger logs error messages on success .. IMPALA-7470: SentryServicePinger logs error messages on success SentryServicePinger checks if Sentry is running by calling a Sentry API to get a list of roles. If Sentry is not yet running, an exception will be thrown. However, the Sentry client implementation will log some error messages when an exception is thrown. For the purpose of SentryServicePinger, this can be too noisy and verbose and may also confuse other developers into thinking it was a failure when starting Sentry. The log messages were muted in IMPALA-6878, however since Impala no longer uses the shaded version of Sentry client in IMPALA-7423, the patch in IMPALA-6878 no longer worked. This patch fixes the muting of Sentry error messages by turning off the log level using the non-shaded version of Sentry Thrift client. Testing: - Manually tested by starting Sentry and did not see any error messages logged into stdout. Change-Id: I75782d23c1cb67564a9265bf3cc94fd590c7b666 Reviewed-on: http://gerrit.cloudera.org:8080/11281 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/test/java/org/apache/impala/testutil/SentryServicePinger.java 1 file changed, 3 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11281 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I75782d23c1cb67564a9265bf3cc94fd590c7b666 Gerrit-Change-Number: 11281 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7470: SentryServicePinger logs error messages on success
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11281 ) Change subject: IMPALA-7470: SentryServicePinger logs error messages on success .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11281 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75782d23c1cb67564a9265bf3cc94fd590c7b666 Gerrit-Change-Number: 11281 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 21 Aug 2018 19:52:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/439/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 1 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 19:06:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7433: reduce logging on executors
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11202 ) Change subject: IMPALA-7433: reduce logging on executors .. Patch Set 4: I don't think so, IMPALA-5845 is a can of worms that would have a separate solution. -- To view, visit http://gerrit.cloudera.org:8080/11202 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c1db44acc6def2b05a4fd032c63716e08cdf5ff Gerrit-Change-Number: 11202 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:46:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7465: fix test kudu scan mem usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11285 ) Change subject: IMPALA-7465: fix test_kudu_scan_mem_usage .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/438/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca Gerrit-Change-Number: 11285 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:44:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7433: reduce logging on executors
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/11202 ) Change subject: IMPALA-7433: reduce logging on executors .. Patch Set 4: Should we also looks at https://issues.apache.org/jira/browse/IMPALA-5845 in this change? -- To view, visit http://gerrit.cloudera.org:8080/11202 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c1db44acc6def2b05a4fd032c63716e08cdf5ff Gerrit-Change-Number: 11202 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:44:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3058/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 3 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:42:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Hello Bharath Vissapragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11286 to look at the new patch set (#3). Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. IMPALA-6844: Fix possible NULL dereference in to_date() builtin If result.ptr allocation fails for some reason inside the StringVal constructor, we still overwrite result.len and continue. This change checks that the StringVal pointer is not NULL before dereferencing it, and returns NULL if it is. Testing: Added a test case of the to_date() function to alloc-fail-init.test to leverage the fault injector --stress_fn_ctx_alloc. Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 --- M be/src/exprs/timestamp-functions-ir.cc M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-init.test 2 files changed, 7 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/11286/3 -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 3 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/440/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 2 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:41:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3057/ -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 2 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:41:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3057/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 2 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:40:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 2 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:39:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11286 ) Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. Patch Set 2: Thanks for fixing -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 2 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:39:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Hello Bharath Vissapragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11286 to look at the new patch set (#2). Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. IMPALA-6844: Fix possible NULL dereference in to_date() builtin If result.ptr allocation fails for some reason inside the StringVal constructor, we still overwrite result.len and continue. This change checks that the the StringVal pointer is not NULL before dereferencing it, and returns NULL if it is. Testing: Added a test case of the to_date() function to alloc-fail-init.test to leverage the fault injector --stress_fn_ctx_alloc. Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 --- M be/src/exprs/timestamp-functions-ir.cc M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-init.test 2 files changed, 7 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/11286/2 -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 2 Gerrit-Owner: Vincent Tran Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-6772: add ORC 1.5.2
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11254 ) Change subject: IMPALA-6772: add ORC 1.5.2 .. Patch Set 1: Verified+1 I did a test build on all platforms earlier. -- To view, visit http://gerrit.cloudera.org:8080/11254 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24430321d0712bc3eda83d77a1a577270eb1e78f Gerrit-Change-Number: 11254 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:33:03 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-6772: add ORC 1.5.2
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11254 ) Change subject: IMPALA-6772: add ORC 1.5.2 .. IMPALA-6772: add ORC 1.5.2 This version has some fixes that are needed for Impala. ORC 1.5.2 got support to consume non-vendored libraries so we can jettison our previous ORC patches and set *_HOME environment variables instead. Update googletest to an official release that is new enough for ORC to build. I don't believe the unofficial release of googletest was consumed by any users of native-toolchain so I removed it rather than trying to generalize the googletest build script to handle both cases. Change-Id: I24430321d0712bc3eda83d77a1a577270eb1e78f Reviewed-on: http://gerrit.cloudera.org:8080/11254 Reviewed-by: Thomas Marshall Tested-by: Tim Armstrong --- M buildall.sh M source/googletest/build.sh M source/orc/build.sh A source/orc/orc-1.5.2-patches/0001-ORC-396-also-look-for-LZ4-libs-in-lib64-subdir.patch 4 files changed, 59 insertions(+), 17 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/11254 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I24430321d0712bc3eda83d77a1a577270eb1e78f Gerrit-Change-Number: 11254 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6844: Fix possible NULL dereference in to date() builtin
Vincent Tran has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11286 Change subject: IMPALA-6844: Fix possible NULL dereference in to_date() builtin .. IMPALA-6844: Fix possible NULL dereference in to_date() builtin If result.ptr allocation fails for some reason inside the StringVal constructor, we still overwrite result.len and continue. This change checks that the the StringVal pointer is not NULL before dereferencing it, and returns NULL if it is. Testing: Added a test case of the to_date() function to alloc-fail-init.test to leverage the fault injector --stress_fn_ctx_alloc. Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 --- M be/src/exprs/timestamp-functions-ir.cc 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/11286/1 -- To view, visit http://gerrit.cloudera.org:8080/11286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I14cfb29a592885bb2f39958c8644f93db5220a68 Gerrit-Change-Number: 11286 Gerrit-PatchSet: 1 Gerrit-Owner: Vincent Tran
[Impala-ASF-CR] IMPALA-7465: fix test kudu scan mem usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11285 ) Change subject: IMPALA-7465: fix test_kudu_scan_mem_usage .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/437/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca Gerrit-Change-Number: 11285 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:32:21 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-6772: add ORC 1.5.2
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11254 ) Change subject: IMPALA-6772: add ORC 1.5.2 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11254 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24430321d0712bc3eda83d77a1a577270eb1e78f Gerrit-Change-Number: 11254 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 21 Aug 2018 18:26:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7449: Fix network throughput calculation of DataStreamSender
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11241 ) Change subject: IMPALA-7449: Fix network throughput calculation of DataStreamSender .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11241 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I344ac76c0a1a49b4da3d37d2c547f3d5051ebe24 Gerrit-Change-Number: 11241 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 21 Aug 2018 18:25:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7449: Fix network throughput calculation of DataStreamSender
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11241 ) Change subject: IMPALA-7449: Fix network throughput calculation of DataStreamSender .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3056/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11241 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I344ac76c0a1a49b4da3d37d2c547f3d5051ebe24 Gerrit-Change-Number: 11241 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 21 Aug 2018 18:25:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7465: fix test kudu scan mem usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11285 ) Change subject: IMPALA-7465: fix test_kudu_scan_mem_usage .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3055/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca Gerrit-Change-Number: 11285 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:10:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7465: fix test kudu scan mem usage
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11285 ) Change subject: IMPALA-7465: fix test_kudu_scan_mem_usage .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca Gerrit-Change-Number: 11285 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 18:09:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7465: fix test kudu scan mem usage
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11285 to look at the new patch set (#2). Change subject: IMPALA-7465: fix test_kudu_scan_mem_usage .. IMPALA-7465: fix test_kudu_scan_mem_usage The issue was that the row batch queue could grow a lot if the consumer was slow. Also add an additional test to exercise the OOM code path in Kudu for completeness. Testing: Added sleep to kudu-scan-node.cc that reproduced the problem. Looped modified test to flush out flakiness. Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca --- M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-mem-usage.test M tests/query_test/test_mem_usage_scaling.py 2 files changed, 13 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/11285/2 -- To view, visit http://gerrit.cloudera.org:8080/11285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca Gerrit-Change-Number: 11285 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7465: fix test kudu scan mem usage
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11285 ) Change subject: IMPALA-7465: fix test_kudu_scan_mem_usage .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca Gerrit-Change-Number: 11285 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 21 Aug 2018 18:08:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7465: fix test kudu scan mem usage
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11285 Change subject: IMPALA-7465: fix test_kudu_scan_mem_usage .. IMPALA-7465: fix test_kudu_scan_mem_usage The issue was that the row batch queue could grow a lot if the consumer was slow. Also add an additional test to exercise the OOM code path in Kudu for completeness. Testing: Added sleep to kudu-scan-node.cc that reproduced the problem. Looped modified test to flush out flakiness. Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca --- M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-mem-usage.test 1 file changed, 11 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/11285/1 -- To view, visit http://gerrit.cloudera.org:8080/11285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic4a95b6b6d96a447df68ef4912a86f1e11f219ca Gerrit-Change-Number: 11285 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-6373: Allow primitive type widening on parquet tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11268 ) Change subject: IMPALA-6373: Allow primitive type widening on parquet tables .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/436/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11268 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If93394b035c64cf6fc5f37b54d29c034cc1f86e4 Gerrit-Change-Number: 11268 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Aug 2018 17:30:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6373: Allow primitive type widening on parquet tables
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11268 ) Change subject: IMPALA-6373: Allow primitive type widening on parquet tables .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/11268/2/be/src/exec/parquet-common.h File be/src/exec/parquet-common.h: http://gerrit.cloudera.org:8080/#/c/11268/2/be/src/exec/parquet-common.h@253 PS2, Line 253: inline int ParquetPlainEncoder::Decode( optional: I would prefer to merge the common functionality somehow. One way is to call from these (and probably some other similar) functions a template function like templateDecodeWithConversion(const uint8_t* buffer, const uint8_t* buffer_end, TO_T* v) that would do the memcpy + *v=dest. Another possibility would be to create a type mapper struct and extend the non-specialized Decode(): it could branch on std::is_same> to decide between memcpy and memcpy + *v=dest. Doing the type mapping is quite simple and more useful then the comment table at line 189 IMO. Here is an example: https://stackoverflow.com/questions/4512757/type-mapping-by-templates -- To view, visit http://gerrit.cloudera.org:8080/11268 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If93394b035c64cf6fc5f37b54d29c034cc1f86e4 Gerrit-Change-Number: 11268 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Aug 2018 17:23:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: IMPALA-7460 part 1: require user to install Paramiko and Fabric
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/11264 ) Change subject: WIP: IMPALA-7460 part 1: require user to install Paramiko and Fabric .. Patch Set 3: > Thanks. I'm considering this still a WIP as I do some testing not > covered by GVO. I'll carry this if I only make trivial changes. This breaks data load, so I'm going to have to iron that out. Expect another non-trivial update. -- To view, visit http://gerrit.cloudera.org:8080/11264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf9010a0387b52c95b7bda5d1d4606eba1008b65 Gerrit-Change-Number: 11264 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Brown Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 21 Aug 2018 17:02:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6373: Allow primitive type widening on parquet tables
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11268 ) Change subject: IMPALA-6373: Allow primitive type widening on parquet tables .. Patch Set 2: (7 comments) http://gerrit.cloudera.org:8080/#/c/11268/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11268/1//COMMIT_MSG@12 PS1, Line 12: tinyint > I think that it would be useful to mention that tinyint/smallint/int are al Done http://gerrit.cloudera.org:8080/#/c/11268/1/be/src/exec/parquet-common.h File be/src/exec/parquet-common.h: http://gerrit.cloudera.org:8080/#/c/11268/1/be/src/exec/parquet-common.h@260 PS1, Line 260: return byte_size; > Decode() should return the number of bytes read from the original buffer, s Thanks for the explanation! I got confused for a while when looking at the existing tests. Done. http://gerrit.cloudera.org:8080/#/c/11268/1/be/src/exec/parquet-common.h@271 PS1, Line 271: byte_siz > Same as line 260. Done http://gerrit.cloudera.org:8080/#/c/11268/1/be/src/exec/parquet-common.h@282 PS1, Line 282: return byte_size; > Same as line 260. Done http://gerrit.cloudera.org:8080/#/c/11268/1/be/src/exec/parquet-plain-test.cc File be/src/exec/parquet-plain-test.cc: http://gerrit.cloudera.org:8080/#/c/11268/1/be/src/exec/parquet-plain-test.cc@113 PS1, Line 113: int decoded_size = ParquetPlainEncoder::Decode( > This function should return the encoded size. Done http://gerrit.cloudera.org:8080/#/c/11268/1/be/src/exec/parquet-plain-test.cc@145 PS1, Line 145: buffer, buffer + expected_byte_size, expected_byte_size, ); > Same as line 113. Done http://gerrit.cloudera.org:8080/#/c/11268/1/testdata/data/README File testdata/data/README: http://gerrit.cloudera.org:8080/#/c/11268/1/testdata/data/README@192 PS1, Line 192: o rows wit > Please add a second row with different values. Done -- To view, visit http://gerrit.cloudera.org:8080/11268 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If93394b035c64cf6fc5f37b54d29c034cc1f86e4 Gerrit-Change-Number: 11268 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Aug 2018 16:53:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6373: Allow primitive type widening on parquet tables
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11268 ) Change subject: IMPALA-6373: Allow primitive type widening on parquet tables .. IMPALA-6373: Allow primitive type widening on parquet tables This patch implements support for primitive type widening on parquet tables. It only supports conversion to those types without any loss of precision. - tinyint (INT32) -> smallint (INT32), int (INT32), bigint (INT64), double (DOUBLE) - smallint (INT32) -> int (INT32), bigint (INT64), double (DOUBLE) - int (INT32) -> bigint (INT64), double (DOUBLE) - float (FLOAT) -> double (DOUBLE) Testing: - Added BE test - Added E2E test - Ran core tests Change-Id: If93394b035c64cf6fc5f37b54d29c034cc1f86e4 --- M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M be/src/exec/parquet-common.h M be/src/exec/parquet-metadata-utils.cc M be/src/exec/parquet-plain-test.cc M testdata/data/README A testdata/data/primitive_type_widening.parquet A testdata/workloads/functional-query/queries/QueryTest/parquet-type-widening.test M tests/query_test/test_scanners.py 9 files changed, 146 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11268/2 -- To view, visit http://gerrit.cloudera.org:8080/11268 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If93394b035c64cf6fc5f37b54d29c034cc1f86e4 Gerrit-Change-Number: 11268 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] docs: typo fix in PARQUET ARRAY RESOLUTION
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11284 ) Change subject: docs: typo fix in PARQUET_ARRAY_RESOLUTION .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/51/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84fcc3f13215879ea4c5bc9737f5188baeaa5749 Gerrit-Change-Number: 11284 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 21 Aug 2018 16:51:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7412: width bucket() function overflows too easily
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11282 ) Change subject: IMPALA-7412: width_bucket() function overflows too easily .. Patch Set 1: Code-Review+1 lgtm The review would have been a bit easier for me if the changes compared to https://gerrit.cloudera.org/#/c/11160/ were highlighted somehow. -- To view, visit http://gerrit.cloudera.org:8080/11282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I68262698144029ef7f54e027e586eaf105f36ab3 Gerrit-Change-Number: 11282 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Tue, 21 Aug 2018 16:47:42 + Gerrit-HasComments: No
[Impala-ASF-CR] docs: typo fix in PARQUET ARRAY RESOLUTION
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11284 ) Change subject: docs: typo fix in PARQUET_ARRAY_RESOLUTION .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/51/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84fcc3f13215879ea4c5bc9737f5188baeaa5749 Gerrit-Change-Number: 11284 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 21 Aug 2018 16:37:50 + Gerrit-HasComments: No
[Impala-ASF-CR] docs: typo fix in PARQUET ARRAY RESOLUTION
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11284 Change subject: docs: typo fix in PARQUET_ARRAY_RESOLUTION .. docs: typo fix in PARQUET_ARRAY_RESOLUTION Change-Id: I84fcc3f13215879ea4c5bc9737f5188baeaa5749 --- M docs/topics/impala_parquet_array_resolution.xml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/11284/1 -- To view, visit http://gerrit.cloudera.org:8080/11284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I84fcc3f13215879ea4c5bc9737f5188baeaa5749 Gerrit-Change-Number: 11284 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 9: (3 comments) Thanks for your response! Looking forward for the next patch set! http://gerrit.cloudera.org:8080/#/c/10950/9/be/src/exprs/string-functions.cc File be/src/exprs/string-functions.cc: http://gerrit.cloudera.org:8080/#/c/10950/9/be/src/exprs/string-functions.cc@96 PS9, Line 96: src_ > Done. Good name. I think 'pos_ ' is shorter. yeah, 'pos_' is also a good name for it. http://gerrit.cloudera.org:8080/#/c/10950/9/be/src/exprs/string-functions.cc@166 PS9, Line 166: values.erase > I think I should make the var name more meaningful. I'd like to rename 'val I was just thinking of huge json docs with big arrays, but might still not worth the effort. I'm OK with it as it is. http://gerrit.cloudera.org:8080/#/c/10950/9/be/src/util/string-util.h File be/src/util/string-util.h: http://gerrit.cloudera.org:8080/#/c/10950/9/be/src/util/string-util.h@53 PS9, Line 53: AdvanceIdentifier > Sure. This function is a slightly modified version of https://github.com/ap Oh I see, I still think it's not a good name for either functions unless it is the widely used term in parsers. I am not a native speaker, but advancing something suggests me 'modifying something', just like std::advance() does: https://en.cppreference.com/w/cpp/iterator/advance -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 21 Aug 2018 16:30:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7470: SentryServicePinger logs error messages on success
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11281 ) Change subject: IMPALA-7470: SentryServicePinger logs error messages on success .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11281 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75782d23c1cb67564a9265bf3cc94fd590c7b666 Gerrit-Change-Number: 11281 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 21 Aug 2018 16:30:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7402: fix DCHECK when releasing reservation in scan
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11283 ) Change subject: IMPALA-7402: fix DCHECK when releasing reservation in scan .. Patch Set 3: Failure looks like it's due to a stale maven cache or something: [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/util/SentryProxy.java:[27,52] package org.apache.sentry.provider.db.service.thrift does not exist -- To view, visit http://gerrit.cloudera.org:8080/11283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Gerrit-Change-Number: 11283 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 16:28:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7402: fix DCHECK when releasing reservation in scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11283 ) Change subject: IMPALA-7402: fix DCHECK when releasing reservation in scan .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/434/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Gerrit-Change-Number: 11283 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 16:23:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7402: fix DCHECK when releasing reservation in scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11283 ) Change subject: IMPALA-7402: fix DCHECK when releasing reservation in scan .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/435/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Gerrit-Change-Number: 11283 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 16:23:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7433: reduce logging on executors
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/11202 ) Change subject: IMPALA-7433: reduce logging on executors .. Patch Set 4: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/11202/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11202/4//COMMIT_MSG@19 PS4, Line 19: I0813 12:10:50.374167 31259 query-state.cc:485] Instance completed. instance_id=fd4ae28bc993236e:27343be10003 #in-flight=1 status=OK At this point the interesting question is often "Which one is the last instance that's missing?". Can we print the instance IDs of the last 3 remaining instances? Or a random, running one every time an instance finishes? http://gerrit.cloudera.org:8080/#/c/11202/4/be/src/service/impala-internal-service.cc File be/src/service/impala-internal-service.cc: http://gerrit.cloudera.org:8080/#/c/11202/4/be/src/service/impala-internal-service.cc@49 PS4, Line 49: VLOG_QUERY << "ExecQueryFInstances():" << " query_id=" Porting this RPC to KRPC (IMPALA-7467) will put a queue between the incoming RPC and the actual fragment instance startup. It might make sense to log both events, but if we only opt for one, then the actual dequeue and startup might be more interesting. It's probably not something to decide now though. -- To view, visit http://gerrit.cloudera.org:8080/11202 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c1db44acc6def2b05a4fd032c63716e08cdf5ff Gerrit-Change-Number: 11202 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 16:19:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7437. LRU caching of partitions in impalad
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11208 ) Change subject: IMPALA-7437. LRU caching of partitions in impalad .. IMPALA-7437. LRU caching of partitions in impalad This changes the CatalogdMetaProvider to use a Guava-based LRU cache. The eviction strategy is currently time-based (1 hour), and it only performs caching of some basic items like partition information, the null-partition-key-value, and table column statistics. It does not cache the table entries themselves, which means that we don't need to do any invalidation propagation via the statestore quite yet. Instead, every query will do an initial fetch of the table metadata in order to know the current version number. That version number is then used as part of the cache key for all further metadata, so when the version number changes, all of the prior cache entries become "unreachable" and effectively evicted. Initially, I attempted to implement this by adding a new MetaProvider implementation that would transparently wrap another MetaProvider implementation (either catalogd-based or direct-from-source). However, I found that I wanted to use catalogd-based implementation details like the version number in the cache key, and trying to abstract this behind an interface wasn't very clear. So, I elected to just embed the caching logic into the CatalogdMetaProvider itself. Note that this patch upgrades the Guava reference in the pom from 11.0.2 to 14.0.1. In fact, I found that Guava 14.0.1 was already leaking onto the classpath by being included in hive-exec.jar, so it was ending up picking one or the other in a somewhat unpredictable fashion. The CacheBuilder class had a small API change between v11 and v14 so I needed to ensure a specific version so that Eclipse and Maven agreed on which version to build against. This includes some basic unit testing and I also verified that some query tests like TPCH pass. Change-Id: I9a57521ad851da605604a1e7c48d3d6627da5df5 Reviewed-on: http://gerrit.cloudera.org:8080/11208 Tested-by: Impala Public Jenkins Reviewed-by: Vuk Ercegovac --- M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java A fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java M impala-parent/pom.xml 6 files changed, 538 insertions(+), 38 deletions(-) Approvals: Impala Public Jenkins: Verified Vuk Ercegovac: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9a57521ad851da605604a1e7c48d3d6627da5df5 Gerrit-Change-Number: 11208 Gerrit-PatchSet: 8 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7402: fix DCHECK when releasing reservation in scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11283 ) Change subject: IMPALA-7402: fix DCHECK when releasing reservation in scan .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/433/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I445d306de0c6bfb71359100de2fdf3cd4326f6d9 Gerrit-Change-Number: 11283 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Aug 2018 16:18:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7436: initial fetch-from-catalogd implementation
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11182 ) Change subject: IMPALA-7436: initial fetch-from-catalogd implementation .. IMPALA-7436: initial fetch-from-catalogd implementation This patch adds a new RPC to the catalogd which allows a client to fetch a partial view of table or database metadata. Various subsets of information can be specified and are sent back in fairly "raw" format. A new MetaProvider implementation is added which uses this API to support granular fetching of metadata into the impalad. The interface had to be reworked in a few ways to support this: - This API uses partition IDs instead of names to specify them. So, the listPartitions API now returns opaque PartitionRefs which are passed back to the MetaProvider when loading more partition details. The new implementation stores the IDs in these refs while the direct-to-HMS implementation just uses names. - The fetching of file descriptors was merged into the loading of other partition metadata. I couldn't think of any cases where we needed to list partition details without also fetching the file descriptors so it simplified things a bit to merge the two. This was a lot easier to implement for CatalogdMetaProvider since the file metadata is stored by partition rather than looked up by a directory as in the previous API. This necessitated moving some of the logic out of LocalFsTable into DirectMetaProvider, so LocalFsTable no longer deals directly with HDFS APIs like FileStatus. - The handling of "default partition" for an unpartitioned table moved into the MetaProvider implementations itself instead of LocalFsTable. This is because the CatalogdProvider sees the "default partition" as a partition that actually has an identifier on the catalogd, whereas the DirectMetaProvider does not. So, now both providers export the "default partition" as a partition like all the others. This patch also starts to address one of the potential semantic risks of partial caching on the impalad. If one query fetches some subset of partitions, then a DDL occurs to change the table metadata, and another query is submitted, we want to ensure that the metadata for the latter query still reads a consistent snapshot. In other words, we need to ensure that the metadata like partition list and table schema come from the same snapshot as the finer-grained metadata like partition contents. In order to implement this, the MetadataProvider API now requires that callers use a 'TableRef' object to specify the table to be read, instead of the dbName/tableName. In the DirectMetaProvider we don't have any convenient version numbers for a table, so the TableRef just encapsulates the naming. In the CatalogdMetaProvider, we additionally store the version number of the table, and then all subsequent requests verify that the version number has not changed. If it detects a concurrent modification, an exception is thrown. In a future patch, I'm planning on having the frontend catch the exception and trigger a "re-plan". Change-Id: If49207fc592b1cc552fbcc7199568b6833f86901 Reviewed-on: http://gerrit.cloudera.org:8080/11182 Tested-by: Impala Public Jenkins Reviewed-by: Vuk Ercegovac --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M common/fbs/CMakeLists.txt M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java A fe/src/main/java/org/apache/impala/catalog/local/InconsistentMetadataFetchException.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalHbaseTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalPartitionSpec.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalView.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M
[Impala-ASF-CR] WIP: IMPALA-7460 part 1: require user to install Paramiko and Fabric
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11264 ) Change subject: WIP: IMPALA-7460 part 1: require user to install Paramiko and Fabric .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/432/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf9010a0387b52c95b7bda5d1d4606eba1008b65 Gerrit-Change-Number: 11264 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Brown Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 21 Aug 2018 16:14:06 + Gerrit-HasComments: No
