[Impala-ASF-CR] PROTOTYPE: Produce minidumps if test metrics are zero fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16084 ) Change subject: PROTOTYPE: Produce minidumps if test_metrics_are_zero fails .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6003/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I909611642c9eb2884fc9a89a9b2b000158e62c3a Gerrit-Change-Number: 16084 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 16 Jun 2020 04:03:20 + Gerrit-HasComments: No
[Impala-ASF-CR] PROTOTYPE: Produce minidumps if test metrics are zero fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16084 ) Change subject: PROTOTYPE: Produce minidumps if test_metrics_are_zero fails .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16084/1/tests/common/impala_service.py File tests/common/impala_service.py: http://gerrit.cloudera.org:8080/#/c/16084/1/tests/common/impala_service.py@158 PS1, Line 158: r flake8: E501 line too long (93 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/16084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I909611642c9eb2884fc9a89a9b2b000158e62c3a Gerrit-Change-Number: 16084 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 16 Jun 2020 04:02:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] PROTOTYPE: Produce minidumps if test metrics are zero fails
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16084 Change subject: PROTOTYPE: Produce minidumps if test_metrics_are_zero fails .. PROTOTYPE: Produce minidumps if test_metrics_are_zero fails This is a change to try to diagnose IMPALA-9842. Hopefully, this could get us a stack of a stuck thread. Change-Id: I909611642c9eb2884fc9a89a9b2b000158e62c3a --- M tests/common/impala_service.py 1 file changed, 22 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/16084/1 -- To view, visit http://gerrit.cloudera.org:8080/16084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I909611642c9eb2884fc9a89a9b2b000158e62c3a Gerrit-Change-Number: 16084 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-9341: Set delegateAdmin to false for REVOKE without GRANT OPTION
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16046 ) Change subject: IMPALA-9341: Set delegateAdmin to false for REVOKE without GRANT OPTION .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5998/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19ff45a5a30293e9c6cf35b22ea4aa5cb10355c9 Gerrit-Change-Number: 16046 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 15 Jun 2020 20:07:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16080 ) Change subject: IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6002/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10cabce2908f1d252b90390978e679d31003e89d Gerrit-Change-Number: 16080 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Jun 2020 01:34:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9341: Set delegateAdmin to false for REVOKE without GRANT OPTION
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16046 ) Change subject: IMPALA-9341: Set delegateAdmin to false for REVOKE without GRANT OPTION .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5998/ -- To view, visit http://gerrit.cloudera.org:8080/16046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19ff45a5a30293e9c6cf35b22ea4aa5cb10355c9 Gerrit-Change-Number: 16046 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 16 Jun 2020 01:10:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16074 ) Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5997/ -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 23:38:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9688: Support create iceberg table by impala
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/15797 )
Change subject: IMPALA-9688: Support create iceberg table by impala
..
Patch Set 22:
(11 comments)
Thanks for bearing with me. I left some comments below. The patch is good to go
from my side if you can address those in this patch (or create follow up JIRAs
for addressing them).
http://gerrit.cloudera.org:8080/#/c/15797/21//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15797/21//COMMIT_MSG@11
PS21, Line 11: create table
> Yes, iceberg supported some dml operation as well. I'm working on IMPALA-97
thanks for the clarification.
http://gerrit.cloudera.org:8080/#/c/15797/21//COMMIT_MSG@18
PS21, Line 18: partition by spec(
: level identity,
: event_time identity,
: event_time hour,
: register_time day
: )
> 1. I refer to the syntax of kudu table DDl: partition by hash/range, and I
Thanks. I noticed that Presto has some support for Iceberg as per
https://prestodb.io/docs/current/release/release-0.123.html but I am not sure
how different or similar it is with this.
http://gerrit.cloudera.org:8080/#/c/15797/21/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/15797/21/fe/src/main/cup/sql-parser.cup@1607
PS21, Line 1607: createIcebergPartitionedLayout
What happens when an invalid transform is provided in the input? This method
throws a TableLoadingException which is a bit weird to see during the Parsing
stage. In case of Kudu we don't have that problem since the KW_HASH and
KW_RANGE have specific keywords defined.
http://gerrit.cloudera.org:8080/#/c/15797/21/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/15797/21/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@256
PS21, Line 256: // schema. Likewise for external Kudu tables, the schema
can be read from Kudu.
: if (getColumnDefs().isEmpty() && getFileFormat() !=
THdfsFileFormat.AVRO
: && getFileFormat() != THdfsFileFormat.KUDU) {
:
Can Iceberg tables have 0 columns? If not, may be add a check here too for
Iceberg.
http://gerrit.cloudera.org:8080/#/c/15797/22/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/15797/22/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@143
PS22, Line 143: getIcebergPartitionSpecs
Is it possible that the user creates a iceberg table without providing a
partition by spec clause? What is the behavior in this case for table loading
code?
I think it would great if you could add some positive and negative tests
similar to AnalyzeKuduDDLTest (I think its okay to do it as a follow up in
separate JIRA).
http://gerrit.cloudera.org:8080/#/c/15797/22/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@268
PS22, Line 268: }
:
: if (getFileFormat() == THdfsFileFormat.ICEBERG) {
: analyzeIcebergFormat();
: }
If you look at how Kudu table analysis is implemented,
tableDef_.analyze(analyzer) method in line 253 makes sure that the if there are
any Kudu options used, the table is indeed a Kudu table. So may be we should do
it here as well for Iceberg specific options.
Also, I don't see the IcebergPartitionSpec itself getting analyzed anywhere.
For example, if there are any column types which are not supported we should
throw analysis exception if user tries to provide such columns. Also, we should
make sure that the partition spec column names exist in the table column names
if that is disallowed. I think it will be cleaner if we call
IcebergPartitionSpec.analyze() method here so that we can plugin any checks in
IcebergPartitionField during the createTableStmt analysis phase. So may be add
something like for the sake of completeness.
for (IcebergPartitionSpec partitionSpec : getIceBergPartitionSpecs) {
partitionSpec.analyze();
}
http://gerrit.cloudera.org:8080/#/c/15797/22/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@579
PS22, Line 579: putGeneratedKuduProperty
Can you please rename this to putGeneratedProperty since this method is shared
by Kudu and Iceberg now?
http://gerrit.cloudera.org:8080/#/c/15797/22/fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
File fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java:
http://gerrit.cloudera.org:8080/#/c/15797/22/fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java@28
PS22, Line 28: //Column source id from PartitionField
[Impala-ASF-CR] IMPALA-9829: Add metrics for Spilling to S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16083 ) Change subject: IMPALA-9829: Add metrics for Spilling to S3 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6331/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I152b9c5339cedabe33f8873a2bbf651aa5dbb914 Gerrit-Change-Number: 16083 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 16 Jun 2020 01:46:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9849: Set halt on error=1 for TSAN builds
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16069 ) Change subject: IMPALA-9849: Set halt_on_error=1 for TSAN builds .. IMPALA-9849: Set halt_on_error=1 for TSAN builds Set halt_on_error to true by default for TSAN builds (we already do this for ASAN builds). This ensures that Impala crashes whenever a TSAN error is detected. IMPALA-9568 accidentally broke this. Testing: * Ran dataload + be tests in a TSAN build Change-Id: I268c338d9194a66b37c3ccd97027e3543d27bea7 Reviewed-on: http://gerrit.cloudera.org:8080/16069 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/common/init.cc 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I268c338d9194a66b37c3ccd97027e3543d27bea7 Gerrit-Change-Number: 16069 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16074 ) Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 5 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 23:47:07 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-9837: Switch to GCC 7.5
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15205 ) Change subject: IMPALA-9837: Switch to GCC 7.5 .. Patch Set 2: Verified+1 I ran a build with the docker framework and used the resulting binaries for Impala, +1 verifying. -- To view, visit http://gerrit.cloudera.org:8080/15205 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0a454c93a72b562ee46912cebbdc82b8b8375d2 Gerrit-Change-Number: 15205 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 23:41:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16055 ) Change subject: IMPALA-9844: Ozone support for load data inpath .. Patch Set 7: Hit IMPALA-9842. Trying again. -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 20:51:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16055 ) Change subject: IMPALA-9844: Ozone support for load data inpath .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5999/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 20:52:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16055 ) Change subject: IMPALA-9844: Ozone support for load data inpath .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5999/ -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Jun 2020 02:01:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8830: Fix executor group assignment of coordinator only queries
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14183 ) Change subject: IMPALA-8830: Fix executor group assignment of coordinator only queries .. Patch Set 7: (2 comments) This looks good overall, had minor comments only. http://gerrit.cloudera.org:8080/#/c/14183/7/be/src/scheduling/cluster-membership-mgr.h File be/src/scheduling/cluster-membership-mgr.h: http://gerrit.cloudera.org:8080/#/c/14183/7/be/src/scheduling/cluster-membership-mgr.h@212 PS7, Line 212: static const ExecutorGroup empty_exec_group_; I don't feel too strongly about this, but it might be better in some ways to have this be a field of ClusterMembershipMgr, so that we don't run the static destructors, etc when the process shuts down. http://gerrit.cloudera.org:8080/#/c/14183/7/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14183/7/be/src/scheduling/scheduler.cc@558 PS7, Line 558: VLOG_QUERY << "Exec at coord is " << (exec_at_coord ? "true" : "false"); While we're here, can we make this VLOG(2) or equivalent? It is kinda spammy when I look at logs. -- To view, visit http://gerrit.cloudera.org:8080/14183 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fe098032744aa20bbbe4faddfc67e7a46ce03d5 Gerrit-Change-Number: 14183 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Jun 2020 00:47:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Bikramjeet Vig has removed a vote on this change. Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-9837: Switch to GCC 7.5
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15205 ) Change subject: IMPALA-9837: Switch to GCC 7.5 .. IMPALA-9837: Switch to GCC 7.5 This changes GCC from version 4.9.2 to 7.5.0. The libstdc++ ABI changed between 4.9.2, so packages compiled from this version onward are not interchangeable with packages compiled previously. This also bumps the version of binutils from 2.26.1 to 2.28. Now that the native toolchain is past the major C++11 ABI change, it should be easier to explore using newer versions of GCC. This adds standalone compilations of GCC 8.3 and GCC 9.2. Change-Id: Ia0a454c93a72b562ee46912cebbdc82b8b8375d2 Reviewed-on: http://gerrit.cloudera.org:8080/15205 Reviewed-by: Tim Armstrong Tested-by: Joe McDonnell --- M buildall.sh M init.sh M source/gcc/build.sh 3 files changed, 14 insertions(+), 3 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/15205 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia0a454c93a72b562ee46912cebbdc82b8b8375d2 Gerrit-Change-Number: 15205 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16074 ) Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6000/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 5 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 23:47:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16080 ) Change subject: IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile .. Patch Set 2: Code-Review+2 (2 comments) Thank Anurag for the quick review! Carry on the +2. http://gerrit.cloudera.org:8080/#/c/16080/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java File fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java: http://gerrit.cloudera.org:8080/#/c/16080/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java@263 PS1, Line 263: // Load all partition ids. This will create a Partit > Nit: Change the line to "Load all partition ids. This will create a Partiti Done http://gerrit.cloudera.org:8080/#/c/16080/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java@265 PS1, Line 265: // Load all partitions. This will create one par > Nit: Change the line to "Load all partitions. This will create one partitio Done -- To view, visit http://gerrit.cloudera.org:8080/16080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10cabce2908f1d252b90390978e679d31003e89d Gerrit-Change-Number: 16080 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Jun 2020 01:23:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9849: Set halt on error=1 for TSAN builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16069 ) Change subject: IMPALA-9849: Set halt_on_error=1 for TSAN builds .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268c338d9194a66b37c3ccd97027e3543d27bea7 Gerrit-Change-Number: 16069 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 22:07:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16080 ) Change subject: IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10cabce2908f1d252b90390978e679d31003e89d Gerrit-Change-Number: 16080 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Jun 2020 01:34:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9829: Add metrics for Spilling to S3
Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16083 Change subject: IMPALA-9829: Add metrics for Spilling to S3 .. IMPALA-9829: Add metrics for Spilling to S3 Three types of metrics are added in disk-io-mgr : 1. impala-server.io-mgr.queue--bash.write-latency, unit: ns 2. impala-server.io-mgr.queue--bash.write-size, unit: Bytes 3. impala-server.io-mgr.queue--bash.write-io-error It is a mock function for spill to s3, more test cases will be added later when it is fully functional. Change-Id: I152b9c5339cedabe33f8873a2bbf651aa5dbb914 --- M be/src/runtime/io/disk-io-mgr-internal.h M be/src/runtime/io/disk-io-mgr.cc M common/thrift/metrics.json 3 files changed, 113 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/16083/1 -- To view, visit http://gerrit.cloudera.org:8080/16083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I152b9c5339cedabe33f8873a2bbf651aa5dbb914 Gerrit-Change-Number: 16083 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 16 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Jun 2020 01:33:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/16074 ) Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Patch Set 4: An unrelated flaky HBase test failed (IMPALA-1995). Running GVO again -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 23:46:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9838: Switch to GCC 7.5.0
Joe McDonnell has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/16045 )
Change subject: IMPALA-9838: Switch to GCC 7.5.0
..
IMPALA-9838: Switch to GCC 7.5.0
This upgrades GCC and libstdc++ to version 7.5.0. There
have been ABI changes since 4.9.2, so this means that
the native-toolchain produced with the new compiler is
not interoperable with one produced by the old compiler.
To allow that transition, IMPALA_TOOLCHAIN_PACKAGES_HOME
is now a subdirectory of IMPALA_TOOLCHAIN
(toolchain-packages-gcc${IMPALA_GCC_VERSION}) to distinguish
it from the old packages.
Some Python packages in the impala-python virtualenv are
compiled using the toolchain GCC and now use the new ABI.
This leads to two changes:
1. When constructing the LD_LIBRARY_PATH for impala-python,
we include the GCC libstdc++ libraries. Otherwise, certain
Python packages that use C++ fail on older OSes like Centos 7.
This fixes IMPALA-9804.
2. Since developers work on various branches, this changes
the virtualenv's directory location to a directory with
the GCC version in the name. This allows the virtualenv
built with GCC 7 to coexist with the current virtualenv
built with GCC 4.9.2. The location for the old virtualenv is
${IMPALA_HOME}/infra/python/env. The new location is
${IMPALA_HOME}/infra/python/env-gcc${IMPALA_GCC_VERSION}. This
required updating several impala-python scripts.
There are various odds-and-ends related to the transition:
1. Due to the small string optimization, the size of std::string
changed, which means that various data structures also changed
in size. This required updating some static asserts.
2. There is a bug in clang-tidy that reports a use-after-free
for some code using std::shared_ptr. Clang is not modeling
the shared_ptr correctly, so it is a false-positive. As a workaround,
this disables the clang-analyzer-cplusplus.NewDelete diagnostic.
3. Various small compilation fixes (includes, etc).
Performance testing:
- Ran single-node performance tests on TPC-H for the following
configurations:
- TPC-H Parquet scale 30 with normal configurations
- TPC-H Parquet scale 30 with codegen disabled
- TPC-H Kudu scale 10
None found any significant regressions. Full results are
posted on the JIRA.
- Ran single-node performance tests on targeted-perf scale 10.
No significant regressions.
- The size of binaries (impalad, etc) is slightly smaller with the new GCC:
GCC 4.9.2 release impalad binary: 545664
GCC 7.5.0 release impalad binary: 539900
- Compilation in DEBUG mode is roughly 15-25% faster
Functional testing:
- Ran core jobs, exhaustive release jobs, UBSAN
Change-Id: Ia0beb2b618ba669c9699f8dbc0c52d1203d004e4
Reviewed-on: http://gerrit.cloudera.org:8080/16045
Reviewed-by: Joe McDonnell
Tested-by: Impala Public Jenkins
---
M .clang-tidy
M be/src/runtime/sorter-internal.h
M be/src/runtime/sorter.cc
M be/src/runtime/thread-resource-mgr.cc
M be/src/util/container-util.h
M bin/impala-config.sh
M bin/impala-flake8
M bin/impala-gcovr
M bin/impala-ipython
M bin/impala-pip
M bin/impala-py.test
M bin/impala-python
M bin/impala-python-common.sh
M bin/impala-shell.sh
M bin/set-pythonpath.sh
M infra/python/bootstrap_virtualenv.py
M tests/comparison/ORACLE.txt
17 files changed, 36 insertions(+), 27 deletions(-)
Approvals:
Joe McDonnell: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/16045
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia0beb2b618ba669c9699f8dbc0c52d1203d004e4
Gerrit-Change-Number: 16045
Gerrit-PatchSet: 7
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16080 ) Change subject: IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6332/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10cabce2908f1d252b90390978e679d31003e89d Gerrit-Change-Number: 16080 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Jun 2020 01:50:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9829: Add metrics for Spilling to S3
Yida Wu has abandoned this change. ( http://gerrit.cloudera.org:8080/16078 ) Change subject: IMPALA-9829: Add metrics for Spilling to S3 .. Abandoned open a new commit -- To view, visit http://gerrit.cloudera.org:8080/16078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ia8d15c691d52af00c51af489cb67386772f3dec4 Gerrit-Change-Number: 16078 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6001/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 16 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Jun 2020 01:33:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jun 2020 23:28:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9341: Set delegateAdmin to false for REVOKE without GRANT OPTION
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16046 ) Change subject: IMPALA-9341: Set delegateAdmin to false for REVOKE without GRANT OPTION .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19ff45a5a30293e9c6cf35b22ea4aa5cb10355c9 Gerrit-Change-Number: 16046 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 15 Jun 2020 20:07:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9709: Remove Impala-lzo from the development environment
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15814 ) Change subject: IMPALA-9709: Remove Impala-lzo from the development environment .. IMPALA-9709: Remove Impala-lzo from the development environment This removes Impala-lzo from the Impala development environment. Impala-lzo is not built as part of the Impala build. The LZO plugin is no longer loaded. LZO tables are not loaded during dataload, and LZO is no longer tested. This removes some obsolete scan APIs that were only used by Impala-lzo. With this commit, Impala-lzo would require code changes to build against Impala. The plugin infrastructure is not removed, and this leaves some LZO support code in place. If someone were to decide to revive Impala-lzo, they would still be able to load it as a plugin and get the same functionality as before. This plugin support may be removed later. Testing: - Dryrun of GVO - Modified TestPartitionMetadataUncompressedTextOnly's test_unsupported_text_compression() to add LZO case Change-Id: I3a4f12247d8872b7e14c9feb4b2c58cfd60d4c0e Reviewed-on: http://gerrit.cloudera.org:8080/15814 Reviewed-by: Bikramjeet Vig Tested-by: Joe McDonnell --- M CMakeLists.txt M be/src/exec/hdfs-plugin-text-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/util/codec.cc M bin/bootstrap_system.sh M bin/clean.sh M bin/impala-config.sh M bin/set-ld-library-path.sh M bin/start-impala-cluster.py M buildall.sh M docker/entrypoint.sh M docker/impala_base/Dockerfile M docker/test-with-docker.py M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsCompression.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java D testdata/bad_text_lzo/bad_text.lzo D testdata/bad_text_lzo/bad_text.lzo.index M testdata/bin/create-load-data.sh M testdata/bin/generate-schema-statements.py M testdata/bin/generate-test-vectors.py M testdata/bin/load_nested.py D testdata/bin/lzo_indexer.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-query/functional-query_dimensions.csv M testdata/workloads/functional-query/functional-query_exhaustive.csv M testdata/workloads/functional-query/queries/DataErrorsTest/hdfs-scan-node-errors.test D testdata/workloads/functional-query/queries/QueryTest/disable-lzo-plugin.test M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test M testdata/workloads/functional-query/queries/QueryTest/unsupported-compression-partitions.test M testdata/workloads/perf-regression/perf-regression_dimensions.csv M testdata/workloads/perf-regression/perf-regression_exhaustive.csv M testdata/workloads/perf-regression/perf-regression_pairwise.csv M testdata/workloads/targeted-perf/targeted-perf_dimensions.csv M testdata/workloads/targeted-perf/targeted-perf_exhaustive.csv M testdata/workloads/targeted-perf/targeted-perf_pairwise.csv M testdata/workloads/targeted-stress/targeted-stress_dimensions.csv M testdata/workloads/targeted-stress/targeted-stress_exhaustive.csv M testdata/workloads/targeted-stress/targeted-stress_pairwise.csv M testdata/workloads/tpcds-unmodified/tpcds-unmodified_dimensions.csv M testdata/workloads/tpcds-unmodified/tpcds-unmodified_exhaustive.csv M testdata/workloads/tpcds-unmodified/tpcds-unmodified_pairwise.csv M testdata/workloads/tpcds/tpcds_dimensions.csv M testdata/workloads/tpcds/tpcds_exhaustive.csv M testdata/workloads/tpcds/tpcds_pairwise.csv M testdata/workloads/tpch/tpch_dimensions.csv M testdata/workloads/tpch/tpch_exhaustive.csv M testdata/workloads/tpch/tpch_pairwise.csv M tests/common/test_dimensions.py M tests/custom_cluster/test_hive_text_codec_interop.py D tests/custom_cluster/test_scanner_plugin.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_scanners_fuzz.py 65 files changed, 88 insertions(+), 346 deletions(-) Approvals: Bikramjeet Vig: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/15814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageTyp
[Impala-ASF-CR] IMPALA-452 Add support for string concatenation operator using || construct Separated "||" and "OR" into different tokens. -OR (KW OR) remains the same. (it creates CompoundPredicate a
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15877 ) Change subject: IMPALA-452 Add support for string concatenation operator using || construct Separated "||" and "OR" into different tokens. -OR (KW_OR) remains the same. (it creates CompoundPredicate and expects two BOOLEAN operands) -|| (KW_LOGICAL_OR) creates CompoundVe .. Patch Set 9: (4 comments) http://gerrit.cloudera.org:8080/#/c/15877/9//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15877/9//COMMIT_MSG@7 PS9, Line 7: IMPALA-452 Add support for string concatenation operator using || construct Separated "||" and "OR" into different tokens. commit message is still weirdly formatted. https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala has some guidelines: Finally, please write a good, clear commit message, with a short, descriptive title and a message that is exactly long enough to explain what the problem was, and how it was fixed. Each should have 72 or fewer characters if possible. The first line should have an empty line after it, and the first line should begin with the ticket(s) addressed, followed by a colon and a space: "IMPALA-1234: ". Docs-only commits should have [DOCS] after the ticket numbers, like "IMPALA-1234: [DOCS] ". Here is an example of a good commit message: http://gerrit.cloudera.org:8080/#/c/15877/9/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/15877/9/be/src/exprs/expr-test.cc@3892 PS9, Line 3892: nit: we don't usually have a space before parentheses here and below. It's probably worth running clang-format on your patches - it automates some of this minor stuff https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65868536 http://gerrit.cloudera.org:8080/#/c/15877/9/fe/src/main/java/org/apache/impala/analysis/CompoundVerticalBarExpr.java File fe/src/main/java/org/apache/impala/analysis/CompoundVerticalBarExpr.java: http://gerrit.cloudera.org:8080/#/c/15877/9/fe/src/main/java/org/apache/impala/analysis/CompoundVerticalBarExpr.java@33 PS9, Line 33: private Expr encapsulatedExpr_; Can you comment that this is initialized during analysis. http://gerrit.cloudera.org:8080/#/c/15877/9/testdata/workloads/functional-query/queries/QueryTest/exprs.test File testdata/workloads/functional-query/queries/QueryTest/exprs.test: http://gerrit.cloudera.org:8080/#/c/15877/9/testdata/workloads/functional-query/queries/QueryTest/exprs.test@3099 PS9, Line 3099: select distinct bool_col || int_col < smallint_col, "TEST" || string_col from functional.alltypes WHERE float_col = double_col || (string_col || 'test') = 'testtest'; Can you reformat this so that it's more readable (i.e. not all one one line). Same for the one below. -- To view, visit http://gerrit.cloudera.org:8080/15877 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f990d56ecb1e18d1b2737e8c5eab0d524edfaf Gerrit-Change-Number: 15877 Gerrit-PatchSet: 9 Gerrit-Owner: Martin Zink Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Martin Zink Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Jun 2020 00:58:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Sahil Takiar has removed a vote on this change. Change subject: IMPALA-9844: Ozone support for load data inpath .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/16080 ) Change subject: IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile .. Patch Set 1: Code-Review+2 (2 comments) Minor nits but LGTM. Feel free to carry forward the +2. http://gerrit.cloudera.org:8080/#/c/16080/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java File fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java: http://gerrit.cloudera.org:8080/#/c/16080/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java@263 PS1, Line 263: // Load all partitions ids. Miss the partition list. Nit: Change the line to "Load all partition ids. This will create a PartitionLists miss." http://gerrit.cloudera.org:8080/#/c/16080/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java@265 PS1, Line 265: // Load all partitions. All of them are missing. Nit: Change the line to "Load all partitions. This will create one partition miss per partition." -- To view, visit http://gerrit.cloudera.org:8080/16080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10cabce2908f1d252b90390978e679d31003e89d Gerrit-Change-Number: 16080 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jun 2020 23:53:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile
Hello Anurag Mantripragada, Vihang Karajgaonkar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16080 to look at the new patch set (#2). Change subject: IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile .. IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile The hits and requests metrics of partitions are overcounted due to using an updated map. This patch fixes it and adds test coverage on partition metrics. Tests - Run CatalogdMetaProviderTest Change-Id: I10cabce2908f1d252b90390978e679d31003e89d --- M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java 2 files changed, 43 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/16080/2 -- To view, visit http://gerrit.cloudera.org:8080/16080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I10cabce2908f1d252b90390978e679d31003e89d Gerrit-Change-Number: 16080 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16055 ) Change subject: IMPALA-9844: Ozone support for load data inpath .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5995/ -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 20:18:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9709: Remove Impala-lzo from the development environment
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15814 ) Change subject: IMPALA-9709: Remove Impala-lzo from the development environment .. Patch Set 7: Verified+1 Top of stack passed GVO, carrying +1 down -- To view, visit http://gerrit.cloudera.org:8080/15814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a4f12247d8872b7e14c9feb4b2c58cfd60d4c0e Gerrit-Change-Number: 15814 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 23:41:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 3): Model QuerySchedule as a protobuf
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15961 ) Change subject: IMPALA-9692 (part 3): Model QuerySchedule as a protobuf .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/15961/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15961/4//COMMIT_MSG@21 PS4, Line 21: This has : been replaced with the ExecParams class > the part I don't fully understand is that doesn't the regular QuerySchedule ignore, offline discussion clarified this http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/scheduling/admission-controller.h@341 PS4, Line 341: std::unique_ptr* schedule_result might be nice to put this in a struct called AdmissionResponse? even if it is just one field, I feel like it makes the code cleaner - the HS2 service follows this Request/Response pattern -- To view, visit http://gerrit.cloudera.org:8080/15961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 Gerrit-Change-Number: 15961 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 15 Jun 2020 23:31:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 15: > Patch Set 15: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5994/ The test failure is unrelated: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/10989/testReport/junit/verifiers.test_verify_metrics/TestValidateMetrics/test_metrics_are_zero/ It's tracked at IMPALA-9842. Rerun the GVO. -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Jun 2020 01:33:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16074 ) Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6330/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 19:22:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9709: Remove Impala-lzo from the development environment
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/15814 ) Change subject: IMPALA-9709: Remove Impala-lzo from the development environment .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a4f12247d8872b7e14c9feb4b2c58cfd60d4c0e Gerrit-Change-Number: 15814 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 19:07:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16074 ) Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 18:38:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16074 ) Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5997/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 18:38:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/16074 ) Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Patch Set 3: Code-Review+2 Carrying over +2 -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 18:38:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/16074 ) Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. Patch Set 3: GVO failed since the log line that prints shutdown signal received was missing, this happened because the impalad shutdown before it could flush the log line. Fixed this by making sure log lines are flushed before exiting the process. -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 18:37:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9739: Fix data race during impala graceful shutdown
Hello Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16074 to look at the new patch set (#3). Change subject: IMPALA-9739: Fix data race during impala graceful shutdown .. IMPALA-9739: Fix data race during impala graceful shutdown When impala does a graceful shutdown, exit() method is called at the end that performs cleanup which interferes with the shutdown signal handling thread spawned during init() and triggers a data race which gets caught by the thread sanitizer build. This patch fixes that by using an _exit() call instead. Testing: Ran the offending test TestGracefulShutdown on a thread sanitizer build and made sure no data race was flagged. Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a --- M be/src/service/impala-server.cc 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/16074/3 -- To view, visit http://gerrit.cloudera.org:8080/16074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I59bb5326791cd877df4711e23979f9bd88e4659a Gerrit-Change-Number: 16074 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16082 ) Change subject: IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types) .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6329/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659 Gerrit-Change-Number: 16082 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Jun 2020 17:36:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9849: Set halt on error=1 for TSAN builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16069 ) Change subject: IMPALA-9849: Set halt_on_error=1 for TSAN builds .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268c338d9194a66b37c3ccd97027e3543d27bea7 Gerrit-Change-Number: 16069 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 16:55:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9849: Set halt on error=1 for TSAN builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16069 ) Change subject: IMPALA-9849: Set halt_on_error=1 for TSAN builds .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5996/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268c338d9194a66b37c3ccd97027e3543d27bea7 Gerrit-Change-Number: 16069 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 16:55:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 3): Model QuerySchedule as a protobuf
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15961 ) Change subject: IMPALA-9692 (part 3): Model QuerySchedule as a protobuf .. Patch Set 4: (11 comments) mostly typos and questions so far, still trying to understand this part of the code http://gerrit.cloudera.org:8080/#/c/15961/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15961/4//COMMIT_MSG@21 PS4, Line 21: This has : been replaced with the ExecParams class the part I don't fully understand is that doesn't the regular QuerySchedule object still exist? and it still has a reference to TExecRequest? so is there a specific reason the ExecParams class needs to exist? http://gerrit.cloudera.org:8080/#/c/15961/4//COMMIT_MSG@31 PS4, Line 31: AdmissionController::ReleaseQuery() and ReleaseQueryBackend() now : take a query id as a parameter instead of a QuerySchedule. the reason we do this is because ReleaseQuery() and ReleaseQueryBackend() will become RPCs into the new admission control service, right? and we don't want to serialize the entire QuerySchedule for each RPC call? http://gerrit.cloudera.org:8080/#/c/15961/4//COMMIT_MSG@43 PS4, Line 43: know nit: typo? http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/runtime/coordinator-backend-state.cc File be/src/runtime/coordinator-backend-state.cc: http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/runtime/coordinator-backend-state.cc@a95 PS4, Line 95: : intentionally removed? http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/runtime/exec-params.h File be/src/runtime/exec-params.h: http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/runtime/exec-params.h@51 PS4, Line 51: Return a references nit: typo http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/runtime/exec-params.cc File be/src/runtime/exec-params.cc: http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/runtime/exec-params.cc@32 PS4, Line 32: exer_equest_ nit: typo http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/runtime/exec-params.cc@32 PS4, Line 32: refernces nit: typo http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/runtime/exec-params.cc@48 PS4, Line 48: plan_exec_info[0].fragments[0] is it worth adding some DCHECKs to make sure both of these vectors have at least one element in them? http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/scheduling/cluster-membership-mgr.cc File be/src/scheduling/cluster-membership-mgr.cc: http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/scheduling/cluster-membership-mgr.cc@344 PS4, Line 344: const UniqueIdPB& backend_id why was this changed to use a backend_id instead of a be_desc? just curious http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/scheduling/cluster-membership-mgr.cc@352 PS4, Line 352: INFO should this ever happen? is there a reason it can't be DFATAL? http://gerrit.cloudera.org:8080/#/c/15961/4/common/protobuf/admission_control_service.proto File common/protobuf/admission_control_service.proto: http://gerrit.cloudera.org:8080/#/c/15961/4/common/protobuf/admission_control_service.proto@39 PS4, Line 39: not typo? -- To view, visit http://gerrit.cloudera.org:8080/15961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 Gerrit-Change-Number: 15961 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 15 Jun 2020 16:54:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types)
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16082 Change subject: IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types) .. IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types) Hive ACID supports row-level DELETE and UPDATE operatations on a table. It achieves it via assigning a unique row-id for each row, and maintinaining two sets of files in a table. The first set is in the base/delta directories, they contain the INSERTed rows. The second set of files are in the delete-delta directories, they contain the DELETEd rows. (UPDATE operations are implemented via DELETE+INSERT.) In the filesystem it looks like e.g.: * full_acid/delta_001_001_/_0 * full_acid/delta_002_002_/_0 * full_acid/delete_delta_003_003_/_0 During scanning we need to return INSERTed rows minus DELETEd rows. This patch implements it by creating an ANTI JOIN between the INSERT and DELETE sets. It is a planner-only modification. Every HDFS SCAN that scans full ACID tables (that also have deleted rows) are converted to two HDFS SCANs, one for the INSERT deltas, and one for the DELETE deltas. Then a LEFT ANTI HASH JOIN with BROADCAST distribution mode is created above them. Later we can add support for other distribution modes if the performance requires it. E.g. if we have too many deleted rows then probably we are better off with PARTITIONED distribution mode. We could estimate the number of deleted rows by sampling the delete delta files. The current patch only works for primitive types. I.e. we cannot select nested data if the table has deleted rows. Testing: * added planner test * added e2e tests Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659 --- M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test M testdata/workloads/functional-query/queries/QueryTest/acid-negative.test A testdata/workloads/functional-query/queries/QueryTest/full-acid-scans.test M tests/query_test/test_acid.py 13 files changed, 708 insertions(+), 88 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/16082/1 -- To view, visit http://gerrit.cloudera.org:8080/16082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659 Gerrit-Change-Number: 16082 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9747: More fine-grained codegen for text file scanners
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16059 ) Change subject: IMPALA-9747: More fine-grained codegen for text file scanners .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6328/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id370193af578ecf23ed3c6bfcc65fec448156fa3 Gerrit-Change-Number: 16059 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Jun 2020 16:29:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9747: More fine-grained codegen for text file scanners
Daniel Becker has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16059 ) Change subject: IMPALA-9747: More fine-grained codegen for text file scanners .. IMPALA-9747: More fine-grained codegen for text file scanners Currently if the materialization of any column cannot be codegen'd because its type is unsupported (e.g. CHAR(N)), the whole codegen is cancelled for the text scanner. This commit adds the function TextConverter::SupportsCodegenWriteSlot that returns whether the given ColumnType is supported. If the type is not supported, HdfsScanner codegens code that calls the interpreted version instead of failing codegen. For other columns codegen is used as usually. Benchmarks: TODO Testing: TODO Change-Id: Id370193af578ecf23ed3c6bfcc65fec448156fa3 --- M be/src/codegen/gen_ir_descriptions.py M be/src/exec/hdfs-scanner-ir.cc M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/text-converter.cc M be/src/exec/text-converter.h 6 files changed, 76 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/16059/5 -- To view, visit http://gerrit.cloudera.org:8080/16059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id370193af578ecf23ed3c6bfcc65fec448156fa3 Gerrit-Change-Number: 16059 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9849: Set halt on error=1 for TSAN builds
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16069 ) Change subject: IMPALA-9849: Set halt_on_error=1 for TSAN builds .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268c338d9194a66b37c3ccd97027e3543d27bea7 Gerrit-Change-Number: 16069 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 15:51:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16055 ) Change subject: IMPALA-9844: Ozone support for load data inpath .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6327/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 15:44:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16055 ) Change subject: IMPALA-9844: Ozone support for load data inpath .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 15:17:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16055 ) Change subject: IMPALA-9844: Ozone support for load data inpath .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5995/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 15:17:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16055 ) Change subject: IMPALA-9844: Ozone support for load data inpath .. Patch Set 6: Code-Review+2 Fixed AnalyzeStmtsTest#TestLoadData -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 15:17:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9844: Ozone support for load data inpath
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16055 to look at the new patch set (#7). Change subject: IMPALA-9844: Ozone support for load data inpath .. IMPALA-9844: Ozone support for load data inpath Add Ozone to the whitelist of URIs supported for 'load data inpath' queries. Removed a workaround in FileSystemUtil#isOzoneFileSystem that required CDP jars by default. Now, we support only CDP and can require the CDP Ozone jars. Testing: * Tested manually against a cluster with Ozone deployed Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 --- M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 3 files changed, 9 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/16055/7 -- To view, visit http://gerrit.cloudera.org:8080/16055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I60e8c9ebc6c5b2a29539fac9797cbda32e3cc257 Gerrit-Change-Number: 16055 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 15: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5994/ -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jun 2020 14:52:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9829: Add metrics for Spilling to S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16078 ) Change subject: IMPALA-9829: Add metrics for Spilling to S3 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6326/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8d15c691d52af00c51af489cb67386772f3dec4 Gerrit-Change-Number: 16078 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 15 Jun 2020 14:03:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9829: Add metrics for Spilling to S3
Yida Wu has posted comments on this change. (
http://gerrit.cloudera.org:8080/16078 )
Change subject: IMPALA-9829: Add metrics for Spilling to S3
..
Patch Set 1:
(18 comments)
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr-test.cc
File be/src/runtime/tmp-file-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr-test.cc@1054
PS1, Line 1054:
metrics_->FindMetricForTesting("tmp-file-mgr.hdfs.write-latency");
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr-test.cc@1090
PS1, Line 1090:
metrics_->FindMetricForTesting("tmp-file-mgr.hdfs.write-latency");
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr-test.cc@1122
PS1, Line 1122:
metrics_->FindMetricForTesting("tmp-file-mgr.hdfs.write-latency");
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc
File be/src/runtime/tmp-file-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@242
PS1, Line 242:metrics->RegisterMetric(new
HistogramMetric(MetricDefs::Get(TMP_FILE_MGR_HDFS_WRITE_LATENCY, ""),
> line too long (105 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@242
PS1, Line 242:metrics->RegisterMetric(new
HistogramMetric(MetricDefs::Get(TMP_FILE_MGR_HDFS_WRITE_LATENCY, ""),
> line has trailing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@245
PS1, Line 245: metrics->RegisterMetric(new
HistogramMetric(MetricDefs::Get(TMP_FILE_MGR_HDFS_WRITE_SIZE, ""),
> line has trailing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@245
PS1, Line 245: metrics->RegisterMetric(new
HistogramMetric(MetricDefs::Get(TMP_FILE_MGR_HDFS_WRITE_SIZE, ""),
> line too long (103 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@249
PS1, Line 249: tmp_dirs_.emplace_back(tmp_path.string(),
tmp_dirs[i].bytes_limit, tmp_dirs[i].bytes_used_metric);
> line has trailing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@249
PS1, Line 249: tmp_dirs_.emplace_back(tmp_path.string(),
tmp_dirs[i].bytes_limit, tmp_dirs[i].bytes_used_metric);
> line too long (106 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@254
PS1, Line 254: metrics->RegisterMetric(new
HistogramMetric(MetricDefs::Get(TMP_FILE_MGR_S3_WRITE_LATENCY, ""),
> line too long (104 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@254
PS1, Line 254: metrics->RegisterMetric(new
HistogramMetric(MetricDefs::Get(TMP_FILE_MGR_S3_WRITE_LATENCY, ""),
> line has trailing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@257
PS1, Line 257: metrics->RegisterMetric(new
HistogramMetric(MetricDefs::Get(TMP_FILE_MGR_S3_WRITE_SIZE, ""),
> line has trailing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@257
PS1, Line 257: metrics->RegisterMetric(new
HistogramMetric(MetricDefs::Get(TMP_FILE_MGR_S3_WRITE_SIZE, ""),
> line too long (101 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@261
PS1, Line 261: tmp_dirs_.emplace_back(tmp_path.string(),
tmp_dirs[i].bytes_limit, tmp_dirs[i].bytes_used_metric);
> line too long (114 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@261
PS1, Line 261: tmp_dirs_.emplace_back(tmp_path.string(),
tmp_dirs[i].bytes_limit, tmp_dirs[i].bytes_used_metric);
> line has trailing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/runtime/tmp-file-mgr.cc@290
PS1, Line 290: LOG(INFO) << "Using scratch directory " <<
scratch_subdir_path.string() << " on "
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/util/hdfs-util.cc
File be/src/util/hdfs-util.cc:
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/util/hdfs-util.cc@76
PS1, Line 76: bool IsSpecificPath(const char* path, const char*
specific_prefix, bool check_default_fs) {
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16078/1/be/src/util/hdfs-util.cc@79
PS1, Line 79: return strncmp(ExecEnv::GetInstance()->default_fs().c_str(),
specific_prefix, prefix_len) == 0;
> line too long (99 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/16078
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8d15c691d52af00c51af489cb67386772f3dec4
Gerrit-Change-Number: 16078
Gerrit-PatchSet: 1
Gerrit
[Impala-ASF-CR] IMPALA-9829: Add metrics for Spilling to S3
Yida Wu has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16078 ) Change subject: IMPALA-9829: Add metrics for Spilling to S3 .. IMPALA-9829: Add metrics for Spilling to S3 Metrics of Spilling to HDFS and S3 are added when the HDFS or S3 path has been passed in scatch_dir, like --scratch_dirs='hdfs://hdfspath, s3a://s3path'. Three types of metrics added in this case: 1. tmp-file-mgr.s3/hdfs.write-latency, unit: ns 2. tmp-file-mgr.s3/hdfs.write-size, unit: Bytes 3. tmp-file-mgr.s3/hdfs.write-io-error It is a mock function for spill to s3, more metrics or test cases might be added later. Also, the remote paths should be verified to be valid later. Limitations: 1. No support for HDFS 'port number' by now, due to the current delimiter ':' may conflict the use of 'port number', the rule of the format might be changed later. 2. Only one HDFS disk and one S3 disk can be supported by now. Change-Id: Ia8d15c691d52af00c51af489cb67386772f3dec4 --- M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M common/thrift/metrics.json 6 files changed, 318 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/16078/2 -- To view, visit http://gerrit.cloudera.org:8080/16078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia8d15c691d52af00c51af489cb67386772f3dec4 Gerrit-Change-Number: 16078 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9838: Switch to GCC 7.5.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16045 ) Change subject: IMPALA-9838: Switch to GCC 7.5.0 .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0beb2b618ba669c9699f8dbc0c52d1203d004e4 Gerrit-Change-Number: 16045 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 10:33:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7533: Cache partitions across table versions in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16081 ) Change subject: IMPALA-7533: Cache partitions across table versions in LocalCatalog .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6325/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I512f735b596bc51d553e6d395d108f49727619ed Gerrit-Change-Number: 16081 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Jun 2020 10:30:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7533: Cache partitions across table versions in LocalCatalog
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16081 Change subject: IMPALA-7533: Cache partitions across table versions in LocalCatalog .. IMPALA-7533: Cache partitions across table versions in LocalCatalog In LocalCatlaog cache, partition metadata is cached with a composed key of table name, table version and partition id. Whenever the table version bumps, e.g. due to comments being changed, all the cached partition values will be unreachable. Following queries have to reload the partitions and cache them with the new table version. Actually, the partition id is an unique id across the whole catalog. It's sufficient to identify the partition. However, there are no partition level invalidations if the partition is modified in-place in catalogd. So we have to include the table version and depend on it. After IMPALA-9778, there are no in-place modifications on partition metadata. We can safely reuse partition meta across table versions in LocalCatalog cache. This patch removes the table name and version in the partition cache key. So metadata of unchanged partitions can be reused when table version bumps. Tests: - Add tests in test_local_catalog.py to verify the partition metadata is reused based on profile metrics. Change-Id: I512f735b596bc51d553e6d395d108f49727619ed --- M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M tests/custom_cluster/test_local_catalog.py 4 files changed, 72 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/16081/1 -- To view, visit http://gerrit.cloudera.org:8080/16081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I512f735b596bc51d553e6d395d108f49727619ed Gerrit-Change-Number: 16081 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-6692: Trigger sort node run before hitting memory limit.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15963 ) Change subject: IMPALA-6692: Trigger sort node run before hitting memory limit. .. Patch Set 11: (4 comments) http://gerrit.cloudera.org:8080/#/c/15963/10/be/src/exec/sort-node.cc File be/src/exec/sort-node.cc: http://gerrit.cloudera.org:8080/#/c/15963/10/be/src/exec/sort-node.cc@199 PS10, Line 199: const ::impala::Status& add_statu We usually simply use "Status status = ... ". Status only has a single pointer to an error message (which is nullptr if it is OK), so there's is no win in using a ptr/reference. http://gerrit.cloudera.org:8080/#/c/15963/10/be/src/runtime/sorter.h File be/src/runtime/sorter.h: http://gerrit.cloudera.org:8080/#/c/15963/10/be/src/runtime/sorter.h@160 PS10, Line 160: sot_rub typo http://gerrit.cloudera.org:8080/#/c/15963/10/be/src/runtime/sorter.h@211 PS10, Line 211: specify nit: specifies http://gerrit.cloudera.org:8080/#/c/15963/10/be/src/runtime/sorter.h@217 PS10, Line 217: /// There are two case where it is necessary to run intermediate run. nit: cases -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 15 Jun 2020 09:54:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5994/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jun 2020 09:46:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6324/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jun 2020 09:08:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6323/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jun 2020 09:04:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 15: (1 comment) Rebased the patch after https://gerrit.cloudera.org/c/16008 is merged. http://gerrit.cloudera.org:8080/#/c/15985/14/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/15985/14/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@690 PS14, Line 690: new ParallelFileMetadataLoader( > line too long (95 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jun 2020 08:41:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Hello Aman Sinha, Anurag Mantripragada, Vihang Karajgaonkar, Todd Lipcon, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15985 to look at the new patch set (#15). Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. IMPALA-9778: Refactor partition modifications in DDL/DMLs After this patch, in DDL/DMLs that update metadata of partitions, instead of updating partitions in place, we always create new ones and use them to replace the existing instances. This is guarded by making HdfsPartition immutable. There are several benefits for this: - HdfsPartition can be shared across table versions. In full catalog update mode, catalog update can ignore unchanged partitions (IMPALA-3234) and send the update in partition granularity. - Aborted DDL/DMLs won't leave partition metadata in a bad shape (e.g. IMPALA-8406), which usually requires invalidation to recover. - Fetch-on-demand coordinators can cache partition meta using the partition id as the key. When table version updates, only metadata of changed partitions need to be reloaded (IMPALA-7533). - In the work of decoupling partitions from tables (IMPALA-3127), we don't need to assign a catalog version to partitions since the partition ids already identify the partitions. However, HdfsPartition is not strictly immutable. Although all its fields are final, some fields are still referencing mutable objects. We need more refactoring to achieve this. This patch focuses on refactoring the DDL/DML code paths. Changes: - Make all fields of HdfsPartition final. Move HdfsPartition constructor logics and all its update methods into HdfsPartition.Builder. - Refactor in-place updates on HdfsPartition to be creating a new one and dropping the old one. HdfsPartition.Builder represents the in-progress modifications. Once all modifications are done, call its build() method to create the new HdfsPartition instance. The old HdfsPartition instance is only replaced at the end of the modifications. - Move the "dirty" marker of HdfsPartition into a map of HdfsTable. It maps from the old partition id to the in-progress partition builder. For "dirty" partitions, we’ll reload its HMS meta and file meta. Tests: - No new tests are added since the existing tests already provide sufficient coverage - Run CORE tests Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java 11 files changed, 819 insertions(+), 512 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/15985/15 -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15985 ) Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. Patch Set 14: (1 comment) http://gerrit.cloudera.org:8080/#/c/15985/14/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/15985/14/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@690 PS14, Line 690: new ParallelFileMetadataLoader(this, partBuilders, validWriteIds_, validTxnList, logPrefix) line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jun 2020 08:37:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9778: Refactor partition modifications in DDL/DMLs
Hello Aman Sinha, Anurag Mantripragada, Vihang Karajgaonkar, Todd Lipcon, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15985 to look at the new patch set (#14). Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs .. IMPALA-9778: Refactor partition modifications in DDL/DMLs After this patch, in DDL/DMLs that update metadata of partitions, instead of updating partitions in place, we always create new ones and use them to replace the existing instances. This is guarded by making HdfsPartition immutable. There are several benefits for this: - HdfsPartition can be shared across table versions. In full catalog update mode, catalog update can ignore unchanged partitions (IMPALA-3234) and send the update in partition granularity. - Aborted DDL/DMLs won't leave partition metadata in a bad shape (e.g. IMPALA-8406), which usually requires invalidation to recover. - Fetch-on-demand coordinators can cache partition meta using the partition id as the key. When table version updates, only metadata of changed partitions need to be reloaded (IMPALA-7533). - In the work of decoupling partitions from tables (IMPALA-3127), we don't need to assign a catalog version to partitions since the partition ids already identify the partitions. However, HdfsPartition is not strictly immutable. Although all its fields are final, some fields are still referencing mutable objects. We need more refactoring to achieve this. This patch focuses on refactoring the DDL/DML code paths. Changes: - Make all fields of HdfsPartition final. Move HdfsPartition constructor logics and all its update methods into HdfsPartition.Builder. - Refactor in-place updates on HdfsPartition to be creating a new one and dropping the old one. HdfsPartition.Builder represents the in-progress modifications. Once all modifications are done, call its build() method to create the new HdfsPartition instance. The old HdfsPartition instance is only replaced at the end of the modifications. - Move the "dirty" marker of HdfsPartition into a map of HdfsTable. It maps from the old partition id to the in-progress partition builder. For "dirty" partitions, we’ll reload its HMS meta and file meta. Tests: - No new tests are added since the existing tests already provide sufficient coverage - Run CORE tests Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java 11 files changed, 818 insertions(+), 512 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/15985/14 -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16080 ) Change subject: IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6322/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10cabce2908f1d252b90390978e679d31003e89d Gerrit-Change-Number: 16080 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jun 2020 07:38:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16080 Change subject: IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile .. IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile The hits and requests metrics of partitions are overcounted due to using an updated map. This patch fixes it and adds test coverage on partition metrics. Tests - Run CatalogdMetaProviderTest Change-Id: I10cabce2908f1d252b90390978e679d31003e89d --- M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java 2 files changed, 43 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/16080/1 -- To view, visit http://gerrit.cloudera.org:8080/16080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I10cabce2908f1d252b90390978e679d31003e89d Gerrit-Change-Number: 16080 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
