[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18141 ) Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10211/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25080628233799aa0b6be18d5a832f1385414501 Gerrit-Change-Number: 18141 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 07:57:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 07:55:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7866/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 07:55:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 07:55:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
pranav.lo...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 6: > Patch Set 5: Code-Review+1 > > (3 comments) Done! -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 07:54:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18263 to look at the new patch set (#6). Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. IMPALA-2: Impala can't resolve json tables created by Hive Impala was using wrong json serde string which was causing the subsequent errors. The error got resolved on correcting the json serde string. Now impala can resolve json tables created by hive and also hive can resolve json tables created in impala. End-to-end tests are also included in tests/metadata/test_hms_integration.py. Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 --- M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java M tests/metadata/test_hms_integration.py 3 files changed, 22 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/18263/6 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 5: Code-Review+1 (3 comments) http://gerrit.cloudera.org:8080/#/c/18263/5/tests/metadata/test_hms_integration.py File tests/metadata/test_hms_integration.py: http://gerrit.cloudera.org:8080/#/c/18263/5/tests/metadata/test_hms_integration.py@675 PS5, Line 675: @pytest.mark.execute_serially I think we don't need this marker. This test doesn't seem to conflict with others. http://gerrit.cloudera.org:8080/#/c/18263/5/tests/metadata/test_hms_integration.py@676 PS5, Line 676: test_desc_table nit: test_desc_json_table http://gerrit.cloudera.org:8080/#/c/18263/5/tests/metadata/test_hms_integration.py@690 PS5, Line 690: nit: remove the two spaces -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 07:46:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/18141 )
Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables
..
Patch Set 14:
(14 comments)
Thank Qifan for the detailed review!
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc
File be/src/util/in-list-filter-ir.cc:
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc@26
PS13, Line 26: if (UNLIKELY(val == nullptr)) {
> UNLIKELY
Done
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc@30
PS13, Line 30: if (UNLIKELY(values_.size() >= entry_limit_ ||
str_values_.size() >= entry_limit_)) {
> UNLIKELY
Done
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc@55
PS13, Line 55: if (UNLIKELY(s->ptr == nullptr)) {
: contains_null_ = true
> nit. should we check null-ness again? See line at 26.
The default constructor of StringValue creates a null 'ptr'. I think we'd
better check this.
https://github.com/apache/impala/blob/fe04c50/be/src/runtime/string-value.h#L51
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc@58
PS13, Line 58: str_total
> nit. Probably should be named as str_total_size_.
Done
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.h
File be/src/util/in-list-filter.h:
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.h@39
PS13, Line 39: InListFilter(ColumnType type, uint32_t entry_limit, bool
contains_null = false);
> Include contains_null and column type here.
Done
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc
File be/src/util/in-list-filter.cc:
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@63
PS13, Line 63: retur
> "return false" here helps with release code.
Done
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@71
PS13, Line 71: if (type.type == TYPE
> nit. it is better to supply the type in the cstr.
Done
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@78
PS13, Line 78:
: InListFilter* InListFilter::Create(const InListFilterPB&
protobuf, ColumnType type,
: uint32_t entry_limit, ObjectPool* pool) {
> nit. probably should inited in the cstr.
Done
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@100
PS13, Line 100: break;
> return null?
Done
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@112
PS13, Line 112:
:
: void InListFilter::ToProtobuf(const InListFilter* filter,
InListFilterPB* protobuf) {
: DCHECK(protobuf != nullptr);
: i
> Other fields that are not copied: type_, contains_null_, str_size_ and ent
Oops, we don't need this method. Removed it.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@142
PS13, Line 142: if (type_ == TYPE_STRING || type_ == TYPE_VARCHAR || type_ ==
TYPE_CHAR) {
> Same comment for Copy method: missing fields.
Done
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@174
PS13, Line 174:
> should handle null case.
Oops! Done.
http://gerrit.cloudera.org:8080/#/c/18141/13/testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-query-options.test
File
testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-query-options.test:
http://gerrit.cloudera.org:8080/#/c/18141/13/testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-query-options.test@787
PS13, Line 787: broadcast
> May repeat this test with partition HJ to verify that in-list filters is no
Done. I change the test to verify distributed plan in order to show the
partitioned join. Otherwise it's getting the single node plan by default.
http://gerrit.cloudera.org:8080/#/c/18141/13/testdata/workloads/functional-query/queries/QueryTest/in_list_filters.test
File testdata/workloads/functional-query/queries/QueryTest/in_list_filters.test:
http://gerrit.cloudera.org:8080/#/c/18141/13/testdata/workloads/functional-query/queries/QueryTest/in_list_filters.test@127
PS13, Line 127:
> may add a test on date column type.
Sure. The ORC date_tbl is corrupted and need to wait for
https://gerrit.cloudera.org/c/18262/ to be merged.
--
To view, visit http://gerrit.cloudera.org:8080/18141
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I25080628233799aa0b6be18d5a832f1385414501
Gerrit-Change-Number: 18141
Gerrit-PatchSet: 14
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Wed, 23 Feb 2022 07:33:09 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Hello Qifan Chen, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18141 to look at the new patch set (#14). Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables .. IMPALA-10898: Add runtime IN-list filters for ORC tables ORC files have optional bloom filter indexes for each column. Since ORC-1.7.0, the C++ reader supports pushing down predicates to skip unreleated RowGroups. The pushed down predicates will be evaludated on file indexes (i.e. statistics and bloom filter indexes). Note that only EQUALS and IN-list predicates can leverage bloom filter indexes. Currently Impala has two kinds of runtime filters: bloom filter and min-max filter. Unfortunately they can't be converted into EQUALS or IN-list predicates. So they can't leverage the file level bloom filter indexes. This patch adds runtime IN-list filters for this purpose. Currently they are generated only for small build side (e.g. #rows <= 1024) of a broadcast join. They will only be applied on ORC tables and be pushed down to the ORC reader(i.e. ORC lib). To avoid exploding the IN-list, if #rows of the build side exceeds the threshold (1024), we set the filter to ALWAYS_TRUE and clear its entry. The threshold can be configured by a new query option, RUNTIME_IN_LIST_FILTER_ENTRY_LIMIT. Evaluating runtime IN-list filters is much slower than evaluating runtime bloom filters due to the current simple implementation (i.e. std::unorder_set) and the lack of codegen. So we disable it at row level. For visibility, this patch addes two counters in the HdfsScanNode: - NumPushedDownPredicates - NumPushedDownRuntimeFilters They reflect the predicates and runtime filters that are pushed down to the ORC reader. Currently, runtime IN-list filters are disabled by default. This patch extends the query option, ENABLED_RUNTIME_FILTER_TYPES, to support a comma separated list of filter types. It defaults to be "BLOOM,MIN_MAX". Add "IN_LIST" in it to enable runtime IN-list filters. Ran perf tests on a 3 instances cluster on my desktop using TPC-DS with scale factor 20. It shows significant improvements in some queries: +---+-+++-++++---++-++ | Workload | Query | File Format| Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval | +---+-+++-++++---++-++ | TPCDS(20) | TPCDS-Q67A | orc / snap / block | 35.07 | 44.01 | I -20.32% | 0.38%| 1.38%| 10| I -25.69% | -3.58 | -45.33 | | TPCDS(20) | TPCDS-Q37 | orc / snap / block | 1.08 | 1.45| I -25.23% | 7.14%| 3.09%| 10| I -34.09% | -3.58 | -12.94 | | TPCDS(20) | TPCDS-Q70A | orc / snap / block | 6.30 | 8.60| I -26.81% | 5.24%| 4.21%| 10| I -36.67% | -3.58 | -14.88 | | TPCDS(20) | TPCDS-Q16 | orc / snap / block | 1.33 | 1.85| I -28.28% | 4.98%| 5.92%| 10| I -39.38% | -3.58 | -12.93 | | TPCDS(20) | TPCDS-Q18A | orc / snap / block | 5.70 | 8.06| I -29.25% | 3.00%| 4.12%| 10| I -40.30% | -3.58 | -19.95 | | TPCDS(20) | TPCDS-Q22A | orc / snap / block | 2.01 | 2.97| I -32.21% | 6.12%| 5.94%| 10| I -47.68% | -3.58 | -14.05 | | TPCDS(20) | TPCDS-Q77A | orc / snap / block | 8.49 | 12.44 | I -31.75% | 6.44%| 3.96%| 10| I -49.71% | -3.58 | -16.97 | | TPCDS(20) | TPCDS-Q75 | orc / snap / block | 7.76 | 12.27 | I -36.76% | 5.01%| 3.87%| 10| I -59.56% | -3.58 | -23.26 | | TPCDS(20) | TPCDS-Q21 | orc / snap / block | 0.71 | 1.27| I -44.26% | 4.56%| 4.24%| 10| I -77.31% | -3.58 | -28.31 | | TPCDS(20) | TPCDS-Q80A | orc / snap / block | 9.24 | 20.42 | I -54.77% | 4.03%| 3.82%| 10| I -123.12% | -3.58 | -40.90 | | TPCDS(20) | TPCDS-Q39-1 | orc / snap / block | 1.07 | 2.26| I -52.74% | * 23.83% * | 2.60%| 10| I -149.68% | -3.58 | -14.43 | | TPCDS(20) | TPCDS-Q39-2 | orc / snap / block | 1.00 | 2.33| I -56.95% | * 19.53% * | 2.07%| 10| I -151.89% | -3.58 | -20.81 | +---+-+++-++++---++-++ "Base Avg" is the avg of the original time. "Avg" is the current time. However, we also see some regressions due to the suboptimal implementation. The follow-up JIRAs will
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18256 ) Change subject: IMPALA-11133: Decode author of a commit with utf8 before printing it .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18256 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Gerrit-Change-Number: 18256 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 23 Feb 2022 07:15:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18256 ) Change subject: IMPALA-11133: Decode author of a commit with utf8 before printing it .. IMPALA-11133: Decode author of a commit with utf8 before printing it We found that compare_branches.py could fail if the author of a commit contains non-ASCII characters because the script attempts to print the field. This patch fixes the problem by explicitly decoding the value of author with the encoding 'utf8'. The commit message is also decoded with 'utf8' to prevent similar problems from happening when there are non-ASCII characters in the commit message. Testing: - Manually verified that we won't get the UnicodeDecodeError after this patch. Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Reviewed-on: http://gerrit.cloudera.org:8080/18256 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/compare_branches.py 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18256 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Gerrit-Change-Number: 18256 Gerrit-PatchSet: 6 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11008: fix incorrect to propagate inferred predicates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18234 ) Change subject: IMPALA-11008: fix incorrect to propagate inferred predicates .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10210/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e64230f6d0c2b9ef1560186ceba349a5920ccdf Gerrit-Change-Number: 18234 Gerrit-PatchSet: 6 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 23 Feb 2022 06:39:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10209/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 06:36:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11008: fix incorrect to propagate inferred predicates
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/18234 ) Change subject: IMPALA-11008: fix incorrect to propagate inferred predicates .. Patch Set 6: (4 comments) http://gerrit.cloudera.org:8080/#/c/18234/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18234/5//COMMIT_MSG@30 PS5, Line 30: To generate the inferred predicate we substitue the non-outer-join > Could you summarize how the bug is fixed? Done http://gerrit.cloudera.org:8080/#/c/18234/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/18234/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@514 PS5, Line 514: > nit: remove ":" Done http://gerrit.cloudera.org:8080/#/c/18234/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@514 PS5, Line 514: he slot id of t > nit: the slot id of t2.id Done http://gerrit.cloudera.org:8080/#/c/18234/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@515 PS5, Line 515: ojNullableEq > nit: can we indicate 'nullable side' in the var name, e.g. ojNullableEqualS Done -- To view, visit http://gerrit.cloudera.org:8080/18234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e64230f6d0c2b9ef1560186ceba349a5920ccdf Gerrit-Change-Number: 18234 Gerrit-PatchSet: 6 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 23 Feb 2022 06:17:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11008: fix incorrect to propagate inferred predicates
Xianqing He has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18234 ) Change subject: IMPALA-11008: fix incorrect to propagate inferred predicates .. IMPALA-11008: fix incorrect to propagate inferred predicates It is incorrect to propagate predicates inferred from equi-join conjuncts into a plan subtree that is on the nullable side of an outer join if the predicate is not null-filtering condition for the nullable side. For example: SELECT * FROM ( SELECT id IS NOT NULL AND col IS NULL AS a FROM ( SELECT A.id, B.col FROM A LEFT JOIN B ON A.id = B.id ) t ) t WHERE a = 1 Before this patch the inferred predicate '(B.id is not null and B.col is null) = 1' is evaluated at the scanner of B. Because the predicate '(A.id is not null and B.col is null) = 1' is not null-filtering condition for B. To generate the inferred predicate we substitue the non-outer-join slots first and use 'isNullableConjunct' to do a more strict check on the conjunct before the final substitution. Tests: - Add plan tests in predicate-propagation.test - Add new query tests to verify the correctness of inferred predicates propagation - Ran the full set of verifications in Impala Public Jenkins Change-Id: I9e64230f6d0c2b9ef1560186ceba349a5920ccdf --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test M testdata/workloads/functional-query/queries/QueryTest/outer-joins.test 3 files changed, 279 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/18234/6 -- To view, visit http://gerrit.cloudera.org:8080/18234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e64230f6d0c2b9ef1560186ceba349a5920ccdf Gerrit-Change-Number: 18234 Gerrit-PatchSet: 6 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He
[Impala-ASF-CR] IMPALA-11008: fix incorrect to propagate inferred predicates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18234 ) Change subject: IMPALA-11008: fix incorrect to propagate inferred predicates .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7865/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e64230f6d0c2b9ef1560186ceba349a5920ccdf Gerrit-Change-Number: 18234 Gerrit-PatchSet: 6 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 23 Feb 2022 06:18:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18263 to look at the new patch set (#5). Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. IMPALA-2: Impala can't resolve json tables created by Hive Impala was using wrong json serde string which was causing the subsequent errors. The error got resolved on correcting the json serde string. Now impala can resolve json tables created by hive and also hive can resolve json tables created in impala. End-to-end tests are also included in tests/metadata/test_hms_integration.py. Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 --- M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java M tests/metadata/test_hms_integration.py 3 files changed, 23 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/18263/5 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10208/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 06:15:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
pranav.lo...@cloudera.com has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. IMPALA-2: Impala can't resolve json tables created by Hive Impala was using wrong json serde string which was causing the subsequent errors. The error got resolved on correcting the json serde string. Now impala can resolve json tables created by hive and also hive can resolve json tables created in impala. End-to-end tests are also included in tests/metadata/test_hms_integration.py. Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 --- M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java M tests/metadata/test_hms_integration.py 3 files changed, 23 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/18263/3 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18257 ) Change subject: IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I568fe59df720ea040be4926812412ba4c1510a26 Gerrit-Change-Number: 18257 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 23 Feb 2022 03:45:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18257 ) Change subject: IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file .. IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file Impala returns "Couldn't skip rows in file" error for old Parquet file written by an old Impala (e.g. Impala 2.5, 2.6) In DEBUG build Impala crashes by a DCHECK: Check failed: num_buffered_values_ > 0 (-1 vs. 0) The problem is that in some old Parquet files there can be a mismatch between 'num_values' in a page and the encoded def/rep levels. There is usually one more def/rep levels encoded in these files. In SkipTopLevelRows() we skipped values based on how many def levels are https://github.com/apache/impala/blob/92ce6fe48e75d7780efe9a275122554e59aac916/be/src/exec/parquet/parquet-column-readers.cc#L1308-L1314 Since there are more def levels than values in some old files, num_buferred_values_ could become negative. This patch also takes the value of num_buferred_values_ into account when calculating 'read_count', so we can deal with such files. With this patch we also include the column name in the "Couldn't skip rows" error message, so in the future it'll be easier to identify the problematic columns. Testing: * added Parquet file written by Impala 2.5 and e2e test for it Change-Id: I568fe59df720ea040be4926812412ba4c1510a26 Reviewed-on: http://gerrit.cloudera.org:8080/18257 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-column-readers.cc M common/thrift/generate_error_codes.py M testdata/data/README A testdata/data/too_many_def_levels.parquet M tests/query_test/test_scanners.py 6 files changed, 28 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I568fe59df720ea040be4926812412ba4c1510a26 Gerrit-Change-Number: 18257 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen
[Impala-ASF-CR] IMPALA-11008: fix incorrect to propagate inferred predicates
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18234 ) Change subject: IMPALA-11008: fix incorrect to propagate inferred predicates .. Patch Set 5: Code-Review+1 (4 comments) http://gerrit.cloudera.org:8080/#/c/18234/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18234/5//COMMIT_MSG@30 PS5, Line 30: Could you summarize how the bug is fixed? http://gerrit.cloudera.org:8080/#/c/18234/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/18234/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@514 PS5, Line 514: : nit: remove ":" http://gerrit.cloudera.org:8080/#/c/18234/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@514 PS5, Line 514: t2.id's slot id nit: the slot id of t2.id http://gerrit.cloudera.org:8080/#/c/18234/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@515 PS5, Line 515: ojEqualSlots nit: can we indicate 'nullable side' in the var name, e.g. ojNullableEqualSlots? Feel free to find a better name. -- To view, visit http://gerrit.cloudera.org:8080/18234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e64230f6d0c2b9ef1560186ceba349a5920ccdf Gerrit-Change-Number: 18234 Gerrit-PatchSet: 5 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 23 Feb 2022 03:21:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [PROTOTYPE] IMPALA-11125: Revisit the contents of impala-minimal-s3a-aws-sdk jar
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18266 ) Change subject: [PROTOTYPE] IMPALA-11125: Revisit the contents of impala-minimal-s3a-aws-sdk jar .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10207/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8092b159d15836d40ef34d390698921216342b5b Gerrit-Change-Number: 18266 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 23 Feb 2022 03:01:50 + Gerrit-HasComments: No
[Impala-ASF-CR] [PROTOTYPE] IMPALA-11125: Revisit the contents of impala-minimal-s3a-aws-sdk jar
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18266 Change subject: [PROTOTYPE] IMPALA-11125: Revisit the contents of impala-minimal-s3a-aws-sdk jar .. [PROTOTYPE] IMPALA-11125: Revisit the contents of impala-minimal-s3a-aws-sdk jar This revisits the list of excluded components for impala-minimal-s3a-aws-sdk.jar. As new components are added in the AWS SDK bundle, this build code gets out of date and doesn't exclude the new things. This adds exclusions for more services in the AWS SDK bundle. It also excludes the models, which are JSON files that don't seem to be in use. Size of impala-minimal-s3a-aws-sdk.jar: Before: 89.7MB After: 13.6MB Testing: - S3 core tests pass Change-Id: I8092b159d15836d40ef34d390698921216342b5b --- M java/shaded-deps/s3a-aws-sdk/pom.xml 1 file changed, 145 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/18266/1 -- To view, visit http://gerrit.cloudera.org:8080/18266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8092b159d15836d40ef34d390698921216342b5b Gerrit-Change-Number: 18266 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18250 ) Change subject: IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail .. Patch Set 4: Code-Review+2 (1 comment) Thanks for digging into this! http://gerrit.cloudera.org:8080/#/c/18250/4/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java File fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java: http://gerrit.cloudera.org:8080/#/c/18250/4/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java@322 PS4, Line 322: rows_from_HMS nit: rowsFromHms. We use CamelCase in FE codes. -- To view, visit http://gerrit.cloudera.org:8080/18250 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c51f76212e1337a7e726097931890c2edab182 Gerrit-Change-Number: 18250 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 02:22:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7861/ -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 23 Feb 2022 02:22:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10049: Include RPC call id in slow RPC logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18243 ) Change subject: IMPALA-10049: Include RPC call_id in slow RPC logs .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7864/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fb5746fa0be575745a8e168405d43115c425389 Gerrit-Change-Number: 18243 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 23 Feb 2022 02:14:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10049: Include RPC call id in slow RPC logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18243 ) Change subject: IMPALA-10049: Include RPC call_id in slow RPC logs .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fb5746fa0be575745a8e168405d43115c425389 Gerrit-Change-Number: 18243 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 23 Feb 2022 02:14:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10049: Include RPC call id in slow RPC logs
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18243 ) Change subject: IMPALA-10049: Include RPC call_id in slow RPC logs .. Patch Set 2: Hit IMPALA-11144 -- To view, visit http://gerrit.cloudera.org:8080/18243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fb5746fa0be575745a8e168405d43115c425389 Gerrit-Change-Number: 18243 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 23 Feb 2022 02:13:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10049: Include RPC call id in slow RPC logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18243 ) Change subject: IMPALA-10049: Include RPC call_id in slow RPC logs .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7860/ -- To view, visit http://gerrit.cloudera.org:8080/18243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fb5746fa0be575745a8e168405d43115c425389 Gerrit-Change-Number: 18243 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 23 Feb 2022 01:53:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 2: (6 comments) http://gerrit.cloudera.org:8080/#/c/18263/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18263/2//COMMIT_MSG@9 PS2, Line 9: nit: no indention here http://gerrit.cloudera.org:8080/#/c/18263/2//COMMIT_MSG@11 PS2, Line 11: . nit: add a space after periods. http://gerrit.cloudera.org:8080/#/c/18263/2//COMMIT_MSG@16 PS2, Line 16: } nit: remove this http://gerrit.cloudera.org:8080/#/c/18263/2/fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java File fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java: http://gerrit.cloudera.org:8080/#/c/18263/2/fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java@77 PS2, Line 77: , I think this causes the build failure. http://gerrit.cloudera.org:8080/#/c/18263/2/tests/metadata/test_hms_integration.py File tests/metadata/test_hms_integration.py: http://gerrit.cloudera.org:8080/#/c/18263/2/tests/metadata/test_hms_integration.py@684 PS2, Line 684: (x int) We should add 'stored as jsonfile' at the end. Otherwise, it's not a json table. http://gerrit.cloudera.org:8080/#/c/18263/2/tests/metadata/test_hms_integration.py@685 PS2, Line 685: % Does it work? Shouldn't it be '+'? 'DESC ' is not a format string. -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 01:38:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 2: > Patch Set 2: > > Build Failed > > https://jenkins.impala.io/job/gerrit-code-review-checks/10206/ : Initial code > review checks failed. See linked job for details on the failure. The clang-tidy job failed: https://jenkins.impala.io/job/clang-tidy-ub1604/18007/ The log fail is too large to download (1.23GB): https://jenkins.impala.io/job/clang-tidy-ub1604/18007/artifact/tidylog.txt You can reproduce the build failure by adding -tidy option to your buildall.sh, e.g. ./buildall.sh -noclean -notests -tidy -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 00:51:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10206/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 00:39:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18256 ) Change subject: IMPALA-11133: Decode author of a commit with utf8 before printing it .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7863/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18256 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Gerrit-Change-Number: 18256 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 23 Feb 2022 00:34:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18256 ) Change subject: IMPALA-11133: Decode author of a commit with utf8 before printing it .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18256 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Gerrit-Change-Number: 18256 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 23 Feb 2022 00:34:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18256 ) Change subject: IMPALA-11133: Decode author of a commit with utf8 before printing it .. Patch Set 4: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/18256/3/bin/compare_branches.py File bin/compare_branches.py: http://gerrit.cloudera.org:8080/#/c/18256/3/bin/compare_branches.py@270 PS3, Line 270: > Thanks for the detailed explanation Laszlo! Good point! -- To view, visit http://gerrit.cloudera.org:8080/18256 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Gerrit-Change-Number: 18256 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 23 Feb 2022 00:34:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11137: Enable proleptic Gregorian Calendar for Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18262 ) Change subject: IMPALA-11137: Enable proleptic Gregorian Calendar for Hive .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10205/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18262 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6be9c9720dd352d6821cdaa6c64d35ba20473bc0 Gerrit-Change-Number: 18262 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Feb 2022 00:05:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18257 ) Change subject: IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I568fe59df720ea040be4926812412ba4c1510a26 Gerrit-Change-Number: 18257 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Tue, 22 Feb 2022 21:00:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18257 ) Change subject: IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7862/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I568fe59df720ea040be4926812412ba4c1510a26 Gerrit-Change-Number: 18257 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Tue, 22 Feb 2022 21:00:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Csaba Ringhofer has removed a vote on this change. Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18264 ) Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping IMPALA-10961 can decrease the size of the files in functional_parquet.alltypes, which led to breaking the test that rely on the file size. Lowered the expected value, so it fits in the current tolerance range of 0,05. See the jira for more detailed analysis Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Reviewed-on: http://gerrit.cloudera.org:8080/18264 Reviewed-by: Csaba Ringhofer Reviewed-by: Fang-Yu Rao Tested-by: Csaba Ringhofer --- M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, approved; Verified Fang-Yu Rao: Looks good to me, but someone else must approve -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18264 ) Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 20:49:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18264 ) Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. Patch Set 2: The test that failed is org.apache.impala.planner.PlannerTest.testResourceRequirements https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/15812/testReport/junit/org.apache.impala.planner/PlannerTest/testResourceRequirements/ It also used functional_parquet.alltypes, so I think that it fails for the same reason as CardinalityTest While it is a legit and related failure, this fix should make builds more green, so I am merging it as it is -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 20:49:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18257 ) Change subject: IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7858/ -- To view, visit http://gerrit.cloudera.org:8080/18257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I568fe59df720ea040be4926812412ba4c1510a26 Gerrit-Change-Number: 18257 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Tue, 22 Feb 2022 20:45:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18264 ) Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7859/ -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 20:38:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7861/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 22 Feb 2022 19:47:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10204/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 22 Feb 2022 19:41:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10203/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 22 Feb 2022 19:41:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 4: Build Successful https://ec2-35-162-169-52.us-west-2.compute.amazonaws.com/job/gerrit-code-review-checks/10200/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 22 Feb 2022 19:37:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10202/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 22 Feb 2022 19:36:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 3: Build Successful https://ec2-35-162-169-52.us-west-2.compute.amazonaws.com/job/gerrit-code-review-checks/10199/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 22 Feb 2022 19:34:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 2: Build Successful https://ec2-35-162-169-52.us-west-2.compute.amazonaws.com/job/gerrit-code-review-checks/10198/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 22 Feb 2022 19:34:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10049: Include RPC call id in slow RPC logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18243 ) Change subject: IMPALA-10049: Include RPC call_id in slow RPC logs .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7860/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fb5746fa0be575745a8e168405d43115c425389 Gerrit-Change-Number: 18243 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 22 Feb 2022 19:18:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10049: Include RPC call id in slow RPC logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18243 ) Change subject: IMPALA-10049: Include RPC call_id in slow RPC logs .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fb5746fa0be575745a8e168405d43115c425389 Gerrit-Change-Number: 18243 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 22 Feb 2022 19:18:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18200/3/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/18200/3/buildall.sh@233 PS3, Line 233: echo "[-debug_noopt] : Debug build without optimizations applied. The regular debug"\ > line too long (91 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 22 Feb 2022 19:17:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18200 to look at the new patch set (#4). Change subject: IMPALA-0: Switch debug builds to use -Og .. IMPALA-0: Switch debug builds to use -Og GCC's -Og applies optimizations that are compatible with being debuggable. It is similar to -O1 and results in a modest speed-up. This modifies the default debug build to use -Og, so it is now more akin to a fastdebug mode. Even though -Og is intended to preserve debuggability, optimization always impacts debuggability and -Og is no exception. To enable the old behavior, this adds a DEBUG_NOOPT build mode that retains the old non-optimized behavior. Using the -debug_noopt flag with buildall.sh enables this behavior. Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e --- M be/CMakeLists.txt M buildall.sh 2 files changed, 28 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/18200/4 -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18200/3/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/18200/3/buildall.sh@233 PS3, Line 233: echo "[-debug_noopt] : Debug build without optimizations applied. The regular debug"\ line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 19:16:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18200 to look at the new patch set (#3). Change subject: IMPALA-0: Switch debug builds to use -Og .. IMPALA-0: Switch debug builds to use -Og GCC's -Og applies optimizations that are compatible with being debuggable. It is similar to -O1 and results in a modest speed-up. This modifies the default debug build to use -Og, so it is now more akin to a fastdebug mode. Even though -Og is intended to preserve debuggability, optimization always impacts debuggability and -Og is no exception. To enable the old behavior, this adds a DEBUG_NOOPT build mode that retains the old non-optimized behavior. Using the -debug_noopt flag with buildall.sh enables this behavior. Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e --- M be/CMakeLists.txt M buildall.sh 2 files changed, 28 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/18200/3 -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18200 ) Change subject: IMPALA-0: Switch debug builds to use -Og .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/18200/2/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/18200/2/buildall.sh@233 PS2, Line 233: echo "[-debug_noopt] : Debug build without optimizations applied. The regular debug"\ line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 19:13:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11110: Switch debug builds to use -Og
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18200 to look at the new patch set (#2). Change subject: IMPALA-0: Switch debug builds to use -Og .. IMPALA-0: Switch debug builds to use -Og GCC's -Og applies optimizations that are compatible with being debuggable. It is similar to -O1 and results in a modest speed-up. This modifies the default debug build to use -Og, so it is now more akin to a fastdebug mode. Even though -Og is intended to preserve debuggability, optimization always impacts debuggability and -Og is no exception. To enable the old behavior, this adds a DEBUG_NOOPT build mode that retains the old non-optimized behavior. Using the -debug_noopt flag with buildall.sh enables this behavior. Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e --- M be/CMakeLists.txt M buildall.sh 2 files changed, 29 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/18200/2 -- To view, visit http://gerrit.cloudera.org:8080/18200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e Gerrit-Change-Number: 18200 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11135: Deflake LEFT ANTI JOIN test case in test spilling.py
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18261 ) Change subject: IMPALA-11135: Deflake LEFT ANTI JOIN test case in test_spilling.py .. Patch Set 1: Code-Review+1 Thanks for the prompt fix Riza! I do not have any other suggestion. -- To view, visit http://gerrit.cloudera.org:8080/18261 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idad9fc6ec6a0ba7fc70e0701e567da7165e40e83 Gerrit-Change-Number: 18261 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 22 Feb 2022 19:05:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10049: Include RPC call id in slow RPC logs
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18243 ) Change subject: IMPALA-10049: Include RPC call_id in slow RPC logs .. Patch Set 1: Code-Review+2 This looks good to me -- To view, visit http://gerrit.cloudera.org:8080/18243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fb5746fa0be575745a8e168405d43115c425389 Gerrit-Change-Number: 18243 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 22 Feb 2022 19:00:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 ) Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10201/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18265 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2 Gerrit-Change-Number: 18265 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Tue, 22 Feb 2022 18:55:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 ) Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns .. Patch Set 1: Build Successful https://ec2-35-162-169-52.us-west-2.compute.amazonaws.com/job/gerrit-code-review-checks/10197/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18265 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2 Gerrit-Change-Number: 18265 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Tue, 22 Feb 2022 18:52:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9433: Improved caching of HdfsFileHandles
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18191 ) Change subject: IMPALA-9433: Improved caching of HdfsFileHandles .. Patch Set 21: (1 comment) http://gerrit.cloudera.org:8080/#/c/18191/21/be/src/runtime/io/handle-cache.inline.h File be/src/runtime/io/handle-cache.inline.h: http://gerrit.cloudera.org:8080/#/c/18191/21/be/src/runtime/io/handle-cache.inline.h@167 PS21, Line 167: // Opening a file handle requires talking to the NameNode so it can take some time. : RETURN_IF_ERROR(accessor_tmp.Get()->Init(hdfs_monitor_)); > Let me double-check my understanding of the threading here: Another question: How does this work if this call fails? Does the entry get removed from the cache? -- To view, visit http://gerrit.cloudera.org:8080/18191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b5c5e9e2b5db2847ab88c41f667c9ca1b03d51a Gerrit-Change-Number: 18191 Gerrit-PatchSet: 21 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 22 Feb 2022 18:47:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9433: Improved caching of HdfsFileHandles
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18191 ) Change subject: IMPALA-9433: Improved caching of HdfsFileHandles .. Patch Set 21: > > (8 comments) > > > > Thank you for taking this on. There is a lot of history here. > > Originally, the file handle cache used a generic structure like > > this: > > https://github.com/apache/impala/blob/branch-2.10.0/be/src/util/lru-cache.h > > > > In my rewrite, I switched it to remove the generic structure. > This > > heads back in the other direction. > > > > I like that you have backend tests for the generic data > structure, > > which is definitely one advantage of that approach. One question > I > > have about moving back to a generic structure is whether we would > > be able to add new customization to the file handle cache case. I > > had been thinking about adding a file structure that could > contain > > additional per-file data and/or stats. Is that possible with the > > new generic structure? > > "had been thinking about adding a file structure that could contain > additional per-file data and/or stat" > > I was thinking about similar things (e.g. caching processed > Parquet/ORC headers), but this seems a somewhat different feature > to me - while we want to cache more than one file handle per file > and apply LRU logic per handle, we want to cache data for a file > only once and apply LRU logic per file. Yeah, it's unclear whether we would ever want to extend the file handle cache to deal with other things. Separate data structures may be cleaner even if it means duplicating filename strings or other things. The file handle cache is pretty unusual in structure and historically we haven't extended it. I don't have any strong objection to a generic structure. I just wanted to think through whether there are any extensions that would end up getting more complicated. For more ordinary caches that don't need duplication, we should be using the cache implementations in be/src/util/cache, because that also gets us different cache eviction policies. -- To view, visit http://gerrit.cloudera.org:8080/18191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b5c5e9e2b5db2847ab88c41f667c9ca1b03d51a Gerrit-Change-Number: 18191 Gerrit-PatchSet: 21 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 22 Feb 2022 18:46:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18265 ) Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/18265/1/tests/query_test/test_runtime_filters.py File tests/query_test/test_runtime_filters.py: http://gerrit.cloudera.org:8080/#/c/18265/1/tests/query_test/test_runtime_filters.py@328 PS1, Line 328: = flake8: E225 missing whitespace around operator -- To view, visit http://gerrit.cloudera.org:8080/18265 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2 Gerrit-Change-Number: 18265 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Tue, 22 Feb 2022 18:34:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18265 Change subject: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns .. IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns Impala crashes on a Parquet file that contains the partition columns. Data files usually don't contain the partition columns, so Impala don't expect to find such columns in the data files. Unfortunately min/max filtering generates a SEGFAULT when the partition column is present in the data files. It happens when FindSkipRangesForPagesWithMinMaxFilters() tries to retrieve the Parquet schema element for a given slot descriptor. When the slot descriptor refers to a partition column, we usually don't find a schema element so we don't try to skip pages. But when the partition column is present in the data file, the code tries to calculate the filtered pages for the column. It uses the column reader object corresponding to the column, but this is NULL for partition columns, hence we get a SEGFAULT. The code shouldn't do anything at the page-level for partition columns, as the data in such columns are the same for the whole file and it is already filtered at a higher level. Testing: * added e2e test Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2 --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M testdata/data/README A testdata/data/partition_col_in_parquet.parquet M tests/query_test/test_runtime_filters.py 4 files changed, 35 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/18265/1 -- To view, visit http://gerrit.cloudera.org:8080/18265 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2 Gerrit-Change-Number: 18265 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18256 ) Change subject: IMPALA-11133: Decode author of a commit with utf8 before printing it .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10200/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18256 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Gerrit-Change-Number: 18256 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 22 Feb 2022 18:20:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18256 ) Change subject: IMPALA-11133: Decode author of a commit with utf8 before printing it .. Patch Set 4: Build Successful https://ec2-35-162-169-52.us-west-2.compute.amazonaws.com/job/gerrit-code-review-checks/10196/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18256 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Gerrit-Change-Number: 18256 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 22 Feb 2022 18:16:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Fang-Yu Rao has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18256 ) Change subject: IMPALA-11133: Decode author of a commit with utf8 before printing it .. IMPALA-11133: Decode author of a commit with utf8 before printing it We found that compare_branches.py could fail if the author of a commit contains non-ASCII characters because the script attempts to print the field. This patch fixes the problem by explicitly decoding the value of author with the encoding 'utf8'. The commit message is also decoded with 'utf8' to prevent similar problems from happening when there are non-ASCII characters in the commit message. Testing: - Manually verified that we won't get the UnicodeDecodeError after this patch. Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d --- M bin/compare_branches.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/18256/4 -- To view, visit http://gerrit.cloudera.org:8080/18256 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Gerrit-Change-Number: 18256 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/18141 )
Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables
..
Patch Set 13:
(18 comments)
Looks very good!
http://gerrit.cloudera.org:8080/#/c/18141/4/be/src/exec/hdfs-orc-scanner.cc
File be/src/exec/hdfs-orc-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/18141/4/be/src/exec/hdfs-orc-scanner.cc@1271
PS4, Line 1271: gumentBuilder* sarg) {
> hmm, I don't think we will use unmerged versions of runtime filters here. f
Yeah, since the focus of the patch is for broadcasting HJ, I think we are okay.
The merge for partitioned HJ is done here
https://github.com/apache/impala/blob/master/be/src/runtime/runtime-filter-bank.cc#L206
and here
https://github.com/apache/impala/blob/master/be/src/runtime/runtime-filter.cc#L52.
We probably should specifically handle in-list case for
https://github.com/apache/impala/blob/master/be/src/runtime/runtime-filter.cc#L52.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc
File be/src/util/in-list-filter-ir.cc:
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc@26
PS13, Line 26: if (val == nullptr) {
UNLIKELY
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc@30
PS13, Line 30: if (values_.size() >= entry_limit_ || str_values_.size() >=
entry_limit_) {
UNLIKELY
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc@55
PS13, Line 55: if (s->ptr == nullptr) {
: contains_null_ = true
nit. should we check null-ness again? See line at 26.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter-ir.cc@58
PS13, Line 58: str_size_
nit. Probably should be named as str_total_size_.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.h
File be/src/util/in-list-filter.h:
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.h@39
PS13, Line 39: InListFilter(uint32_t entry_limit) : always_true_(false),
entry_limit_(entry_limit) {}
Include contains_null and column type here.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc
File be/src/util/in-list-filter.cc:
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@63
PS13, Line 63: break
"return false" here helps with release code.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@71
PS13, Line 71: filter->type_ = type;
nit. it is better to supply the type in the cstr.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@78
PS13, Line 78: filter->type_ = type;
: filter->contains_null_ = protobuf.contains_null();
: filter->always_true_ = protobuf.always_true();
nit. probably should inited in the cstr.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@100
PS13, Line 100: break;
return null?
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@112
PS13, Line 112: out->set_always_true(in.always_true());
: for (ColumnValuePB in_value : in.value()) {
: ColumnValuePB* out_value = out->add_value();
: *out_value = in_value;
: }
Other fields that are not copied: type_, contains_null_, str_size_ and
entry_limit_.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@142
PS13, Line 142: }
Same comment for Copy method: missing fields.
http://gerrit.cloudera.org:8080/#/c/18141/13/be/src/util/in-list-filter.cc@174
PS13, Line 174: ']'
should handle null case.
http://gerrit.cloudera.org:8080/#/c/18141/8/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/18141/8/common/thrift/ImpalaService.thrift@725
PS8, Line 725: RUNTIME_IN_LIST_FILTER_ENTRY_LIMIT
> I think having the RUNTIME prefix is consistent with existing options, e.g.
Done
http://gerrit.cloudera.org:8080/#/c/18141/4/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java
File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java:
http://gerrit.cloudera.org:8080/#/c/18141/4/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@394
PS4, Line 394: o
> I added a check for the inner side started from patch set 5: https://gerrit
Okay. Thanks.
http://gerrit.cloudera.org:8080/#/c/18141/8/testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
File testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test:
http://gerrit.cloudera.org:8080/#/c/18141/8/testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test@8
PS8, Line 8: 3.44K
> Sorry, these should not be introduced. I replace the test files so got thes
Great. Thanks a lot for taking care if it.
I wonder if we should also check that estimated memory
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18264 ) Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. Patch Set 2: Code-Review+1 Thanks for the prompt analysis and fix! I do not have any additional comment. -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 16:58:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18264 ) Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10199/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 16:14:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18264 ) Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. Patch Set 2: Build Successful https://ec2-35-162-169-52.us-west-2.compute.amazonaws.com/job/gerrit-code-review-checks/10195/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 16:13:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18264 ) Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 15:56:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18264 Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping IMPALA-10961 can decrease the size of the files in functional_parquet.alltypes, which led to breaking the test that rely on the file size. Lowered the expected value, so it fits in the current tolerance range of 0,05. See the jira for more detailed analysis Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 --- M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/18264/2 -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao
[Impala-ASF-CR] IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18264 ) Change subject: IMPALA-11144: fix testAggregationNodeGroupByCardinalityCapping .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7859/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7956db444549c02fab3e56cb8bd281535f679776 Gerrit-Change-Number: 18264 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 15:57:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18250 ) Change subject: IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10198/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18250 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c51f76212e1337a7e726097931890c2edab182 Gerrit-Change-Number: 18250 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 15:40:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18250 ) Change subject: IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail .. Patch Set 4: Build Successful https://ec2-35-162-169-52.us-west-2.compute.amazonaws.com/job/gerrit-code-review-checks/10194/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18250 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c51f76212e1337a7e726097931890c2edab182 Gerrit-Change-Number: 18250 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 22 Feb 2022 15:39:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18257 ) Change subject: IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file .. Patch Set 2: Code-Review+2 +2 to carry Csaba's +1. -- To view, visit http://gerrit.cloudera.org:8080/18257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I568fe59df720ea040be4926812412ba4c1510a26 Gerrit-Change-Number: 18257 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Tue, 22 Feb 2022 15:25:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18257 ) Change subject: IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7858/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I568fe59df720ea040be4926812412ba4c1510a26 Gerrit-Change-Number: 18257 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Tue, 22 Feb 2022 15:27:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18257 ) Change subject: IMPALA-11134: Impala returns "Couldn't skip rows in file" error for old Parquet file .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I568fe59df720ea040be4926812412ba4c1510a26 Gerrit-Change-Number: 18257 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Tue, 22 Feb 2022 15:27:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail
Qifan Chen has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18250 ) Change subject: IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail .. IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail This patch addresses the potential row count over-estimation against HBase tables by capping the estimation by the row count when available from HMS. Testing: 1. Core test [TBD] Change-Id: I11c51f76212e1337a7e726097931890c2edab182 --- M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java 1 file changed, 6 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18250/4 -- To view, visit http://gerrit.cloudera.org:8080/18250 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I11c51f76212e1337a7e726097931890c2edab182 Gerrit-Change-Number: 18250 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9433: Improved caching of HdfsFileHandles
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18191 ) Change subject: IMPALA-9433: Improved caching of HdfsFileHandles .. Patch Set 21: > (8 comments) > > Thank you for taking this on. There is a lot of history here. > Originally, the file handle cache used a generic structure like > this: > https://github.com/apache/impala/blob/branch-2.10.0/be/src/util/lru-cache.h > > In my rewrite, I switched it to remove the generic structure. This > heads back in the other direction. > > I like that you have backend tests for the generic data structure, > which is definitely one advantage of that approach. One question I > have about moving back to a generic structure is whether we would > be able to add new customization to the file handle cache case. I > had been thinking about adding a file structure that could contain > additional per-file data and/or stats. Is that possible with the > new generic structure? "had been thinking about adding a file structure that could contain additional per-file data and/or stat" I was thinking about similar things (e.g. caching processed Parquet/ORC headers), but this seems a somewhat different feature to me - while we want to cache more than one file handle per file and apply LRU logic per handle, we want to cache data for a file only once and apply LRU logic per file. -- To view, visit http://gerrit.cloudera.org:8080/18191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b5c5e9e2b5db2847ab88c41f667c9ca1b03d51a Gerrit-Change-Number: 18191 Gerrit-PatchSet: 21 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 22 Feb 2022 11:22:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Laszlo Gaal has posted comments on this change. (
http://gerrit.cloudera.org:8080/18256 )
Change subject: IMPALA-11133: Decode author of a commit with utf8 before
printing it
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18256/3/bin/compare_branches.py
File bin/compare_branches.py:
http://gerrit.cloudera.org:8080/#/c/18256/3/bin/compare_branches.py@270
PS3, Line 270: msg
I would actually suggest decoding the "msg" field as well: it is free text
coming from (former) user input, so it can also contain non-ASCII characters,
e.g. smart quotes in earlier problems that led to earlier patches to this line.
Another solution could be to explicitly encode each input commit message field
in L147 (changing t.strip() to t.decode('utf-8').strip() ), but that would
require checking the further data flow for the "commit_hash" field. OTOH the
commit hash is guaranteed to comtain only hex digits, so implicit
ASCII->Unicode and reverse transofrmations should not cause any problems.
--
To view, visit http://gerrit.cloudera.org:8080/18256
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d
Gerrit-Change-Number: 18256
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 22 Feb 2022 11:16:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11133: Decode author of a commit with utf8 before printing it
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18256 ) Change subject: IMPALA-11133: Decode author of a commit with utf8 before printing it .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7857/ -- To view, visit http://gerrit.cloudera.org:8080/18256 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb03b0937a994db2bf08e4199574d04f7fb99f5d Gerrit-Change-Number: 18256 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 22 Feb 2022 10:59:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10197/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 22 Feb 2022 09:50:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10193/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 22 Feb 2022 09:49:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
pranav.lo...@cloudera.com has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. IMPALA-2: Impala can't resolve json tables created by Hive Impala was using wrong json serde string which was causing the subsequent errors.The error got resolved on correcting the json serde string.Now impala can resolve json tables created by hive and also hive can resolve json tables created in impala. End-to-end tests are also included in tests/metadata/test_hms_integration.py. } Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 --- M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java M tests/metadata/test_hms_integration.py 3 files changed, 23 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/18263/2 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10196/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 22 Feb 2022 08:47:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10192/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 22 Feb 2022 08:46:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 1: (15 comments) http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py File tests/metadata/test_hms_integration.py: http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@676 PS1, Line 676: , flake8: E231 missing whitespace after ',' http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@678 PS1, Line 678: w flake8: E112 expected an indented block http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@678 PS1, Line 678: h flake8: E999 IndentationError: expected an indented block http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@678 PS1, Line 678: flake8: E501 line too long (102 > 90 characters) http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@679 PS1, Line 679: e flake8: E112 expected an indented block http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@679 PS1, Line 679: = flake8: E225 missing whitespace around operator http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@679 PS1, Line 679: / flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@680 PS1, Line 680: / flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@683 PS1, Line 683: w flake8: E112 expected an indented block http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@683 PS1, Line 683: , flake8: E231 missing whitespace after ',' http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@683 PS1, Line 683: b flake8: E501 line too long (99 > 90 characters) http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@684 PS1, Line 684: e flake8: E112 expected an indented block http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@684 PS1, Line 684: / flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@685 PS1, Line 685: flake8: E202 whitespace before ')' http://gerrit.cloudera.org:8080/#/c/18263/1/tests/metadata/test_hms_integration.py@686 PS1, Line 686: / flake8: E226 missing whitespace around arithmetic operator -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 22 Feb 2022 08:43:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
pranav.lo...@cloudera.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18263 Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. IMPALA-2: Impala can't resolve json tables created by Hive Impala was using wrong json serde string which was causing the subsequent errors.The error got resolved on correcting the json serde string.Now impala can resolve json tables created by hive and also hive can resolve json tables created in impala. End-to-end tests are also included in tests/metadata/test_hms_integration.py. } Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 --- M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java M tests/metadata/test_hms_integration.py 3 files changed, 16 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/18263/1 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Quanlong Huang
