kasakrisz opened a new pull request #1981: URL: https://github.com/apache/hive/pull/1981
### What changes were proposed in this pull request? When transforming plan of materialized view rebuild to incremental rebuild in case of the view definition has aggregate: * Instead of checking the aggregate key is null check the aggregated columns are null or not. Example: ``` HiveProject($f0=[$3], $f1=[$4], $f2=[CASE(IS NULL($2), $5, +($5, $2))]) ``` where `$2` is coming from the Materialized view and `$5` is coming from the delta result set (The rows inserted after the last MV refresh) * When transforming the `newAST` generated from the CBO plan from an insert overwrite plan to a multi insert plan by `CalcitePlanner.fixUpASTAggregateIncrementalRebuild` replace equality operators with null safe equality operators in join condition * The CBO plan contains a Filter on top of the MV scan checking all the aggregate key columns should be not null. Remove this in `fixUpASTAggregateIncrementalRebuild` since we need all rows from the view. * Split the result of the Right outer join to the insert and update branches of the "multi insert statement" plan by checking that the RowId coming from the materialized view is null or not. ### Why are the changes needed? Rows with null aggregate keys was not handled by incremental MV rebuild and it could lead to data corruption. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` vn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver -Dqfile=materialized_view_create_rewrite_4.q,materialized_view_create_rewrite_nulls.q -pl itests/qtest -Pitests ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
