Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 13:
Fix P13 and P14.
P13:
insert into table test_sort_by_zorder.t partition(year, month) /*+ noclustered
*/
select id, int_col, bool_col, year, month from functional.alltypes
10381 Actual does not match expected result:
10382 WRITE TO HDFS [test_sort_by_zorder.t, OVERWRITE=false,
PARTITION-KEYS=(`year`,`month`)] <=== Missing sort. Resolved by copy
sortExpr_ in InsertStmt’s copy cstr
10383
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10384 | partitions=24
10385 |
10386 01:EXCHANGE [HASH(`year`,`month`)]
10387 |
10388 00:SCAN HDFS [functional.alltypes]
10389 HDFS partitions=24/24 files=24 size=478.45KB
10390 row-size=0B cardinality=7.30K
10391
10392 Expected:
10393 WRITE TO HDFS [test_sort_by_zorder.t, OVERWRITE=false,
PARTITION-KEYS=(year,month)]
10394 | partitions=24
10395 |
10396 02:SORT
10397 | order by: LEXICAL: year ASC NULLS LAST, month ASC NULLS LAST, ZORDER:
int_col, bool_col
10398 | row-size=17B cardinality=7.30K
10399 |
10400 01:EXCHANGE [HASH(`year`,`month`)]
P14:
set max_fs_writers=2;
set explain_level=2;
insert /* +SHUFFLE */ into unpartitioned_table select int_col from
functional_parquet.alltypes
14214
14215 Actual does not match expected result:
14216 F01:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
resolved by adding "maxTableSinks_ = other.maxTableSinks_;” to InsertStmt’s
copy cstr
14217 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
14218 | Per-Host Resources: mem-estimate=57.28KB mem-reservation=0B
thread-reservation=1
14219 WRITE TO HDFS [test_hdfs_insert_writer_limit.unpartitioned_table,
OVERWRITE=false]
14220 | partitions=1
14221 | output exprs: int_col
14222 | mem-estimate=16.64KB mem-reservation=0B thread-reservation=0
14223 |
14224 01:EXCHANGE [RANDOM]
14225 | mem-estimate=40.64KB mem-reservation=0B thread-reservation=0
14226 | tuple-ids=0 row-size=4B cardinality=12.78K
14227 | in pipelines: 00(GETNEXT)
14228 |
14229 F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
14230 Per-Host Resources: mem-estimate=16.00MB mem-reservation=16.00KB
thread-reservation=2
14231 00:SCAN HDFS [functional_parquet.alltypes, RANDOM]
14232 HDFS partitions=24/24 files=24 size=200.89KB
14233 stored statistics:
14234 table: rows=unavailable size=unavailable
14235 partitions: 0/24 rows=12.78K
14236 columns: unavailable
14237 extrapolated-rows=disabled max-scan-range-rows=unavailable
14238 mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=1
14239 tuple-ids=0 row-size=0B cardinality=12.78K
14240 in pipelines: 00(GETNEXT)
14241
14242 Expected:
14243 F01:PLAN FRAGMENT [RANDOM] hosts=2 instances=2
14244 | Per-Host Resources: mem-estimate=65.68KB mem-reservation=0B
thread-reservation=1
14245 WRITE TO HDFS [test_hdfs_insert_writer_limit.unpartitioned_table,
OVERWRITE=false]
14246 | partitions=1
14247 | output exprs: int_col
14248 | mem-estimate=25.01KB mem-reservation=0B thread-reservation=0
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 13
Gerrit-Owner: Qifan Chen <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Comment-Date: Fri, 10 Dec 2021 01:55:34 +0000
Gerrit-HasComments: No
