[
https://issues.apache.org/jira/browse/IMPALA-9081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961496#comment-16961496
]
Tim Armstrong commented on IMPALA-9081:
---------------------------------------
Ok I know what's going on here. In a couple of these tests we flip it from
being a test to a non-test environment. This actually affects whether partition
stats are updated via isStoredInImpaladCatalogCache(). So the loaded table
differs depending on whether it was loaded when that flag was toggled.
{noformat}
@Test
public void testMtDopValidationWithHDFSNumRowsEstDisabled() {
// Tests that queries supported with mt_dop > 0 produce a parallel plan, or
// throw a NotImplementedException otherwise (e.g. plan has a distributed
join).
TQueryOptions options = defaultQueryOptions();
options.setMt_dop(3);
options.setDisable_hdfs_num_rows_estimate(true);
try {
// Temporarily unset the test env such that unsupported queries with
mt_dop > 0
// throw an exception. Those are otherwise allowed for testing parallel
plans.
RuntimeEnv.INSTANCE.setTestEnv(false);
runPlannerTestFile("mt-dop-validation", options);
} finally {
RuntimeEnv.INSTANCE.setTestEnv(true);
}
}
{noformat}
I think we should just leave this alone since these validation tests will
eventually become redundant
> testMtDopValidationWithHDFSNumRowsEstDisabled appears flaky
> -----------------------------------------------------------
>
> Key: IMPALA-9081
> URL: https://issues.apache.org/jira/browse/IMPALA-9081
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Thomas Tauber-Marshall
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> Section PLAN of query:
> insert into functional_parquet.alltypes partition(year,month)
> select * from functional_parquet.alltypessmall
> Actual does not match expected result:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=1.01GB mem-reservation=12.09MB
> thread-reservation=1
> WRITE TO HDFS [functional_parquet.alltypes, OVERWRITE=false,
> PARTITION-KEYS=(year,month)]
> | partitions=unavailable
> ^^^^^^^^^^^^^^^^^^^^^^^^^
> | output exprs: id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month
> | mem-estimate=1.00GB mem-reservation=0B thread-reservation=0
> |
> 01:SORT
> | order by: year ASC NULLS LAST, month ASC NULLS LAST
> | mem-estimate=12.00MB mem-reservation=12.00MB spill-buffer=2.00MB
> thread-reservation=0
> | tuple-ids=2 row-size=80B cardinality=unavailable
> | in pipelines: 01(GETNEXT), 00(OPEN)
> |
> 00:SCAN HDFS [functional_parquet.alltypessmall]
> HDFS partitions=4/4 files=4 size=14.76KB
> stored statistics:
> table: rows=unavailable size=unavailable
> partitions: 0/4 rows=unavailable
> columns: unavailable
> extrapolated-rows=disabled max-scan-range-rows=unavailable
> mem-estimate=16.00MB mem-reservation=88.00KB thread-reservation=0
> tuple-ids=0 row-size=80B cardinality=unavailable
> in pipelines: 00(GETNEXT)
> Expected:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=1.01GB mem-reservation=12.09MB
> thread-reservation=1
> WRITE TO HDFS [functional_parquet.alltypes, OVERWRITE=false,
> PARTITION-KEYS=(year,month)]
> | partitions=4
> | output exprs: id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month
> | mem-estimate=1.00GB mem-reservation=0B thread-reservation=0
> |
> 01:SORT
> | order by: year ASC NULLS LAST, month ASC NULLS LAST
> | mem-estimate=12.00MB mem-reservation=12.00MB spill-buffer=2.00MB
> thread-reservation=0
> | tuple-ids=2 row-size=80B cardinality=unavailable
> | in pipelines: 01(GETNEXT), 00(OPEN)
> |
> 00:SCAN HDFS [functional_parquet.alltypessmall]
> HDFS partitions=4/4 files=4 size=14.51KB
> stored statistics:
> table: rows=unavailable size=unavailable
> partitions: 0/4 rows=unavailable
> columns missing stats: id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col
> extrapolated-rows=disabled max-scan-range-rows=unavailable
> mem-estimate=16.00MB mem-reservation=88.00KB thread-reservation=0
> tuple-ids=0 row-size=80B cardinality=unavailable
> in pipelines: 00(GETNEXT)
> Verbose plan:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=1.01GB mem-reservation=12.09MB
> thread-reservation=1
> WRITE TO HDFS [functional_parquet.alltypes, OVERWRITE=false,
> PARTITION-KEYS=(year,month)]
> | partitions=unavailable
> | output exprs: id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month
> | mem-estimate=1.00GB mem-reservation=0B thread-reservation=0
> |
> 01:SORT
> | order by: year ASC NULLS LAST, month ASC NULLS LAST
> | mem-estimate=12.00MB mem-reservation=12.00MB spill-buffer=2.00MB
> thread-reservation=0
> | tuple-ids=2 row-size=80B cardinality=unavailable
> | in pipelines: 01(GETNEXT), 00(OPEN)
> |
> 00:SCAN HDFS [functional_parquet.alltypessmall]
> HDFS partitions=4/4 files=4 size=14.76KB
> stored statistics:
> table: rows=unavailable size=unavailable
> partitions: 0/4 rows=unavailable
> columns: unavailable
> extrapolated-rows=disabled max-scan-range-rows=unavailable
> mem-estimate=16.00MB mem-reservation=88.00KB thread-reservation=0
> tuple-ids=0 row-size=80B cardinality=unavailable
> in pipelines: 00(GETNEXT)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
