[
https://issues.apache.org/jira/browse/FLINK-31062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-31062:
-------------------------------
Description:
This issue aims to verify the FLINK-31062: New dynamic partition pruning
strategy to support more dpp patterns . This is an extension work of FLIP-248 :
[https://cwiki.apache.org/confluence/display/FLINK/FLIP-248%3A+Introduce+dynamic+partition+pruning]
to enable more query patterns support dynamic partition pruning, like
subQuery, Union etc.
This work can be verified in SQL client after we build the flink-disk package.
# create a partition table and a non-partition table (only hive connector is
supported now), and then insert some data.
# you can verify this work by checking the query plan by executing 'explain'.
Most basic dpp patterns have been verified in Flink-1.16 [1]. So we only need
to focus on verifying the newly supported patterns:
* Test the case that the dim table and the fact table are not adjacent in the
join order: disable join reorder by setting config:
'table.optimizer.join-reorder-enabled = false', and write a query like the
test 'DynamicPartitionPruningProgramTest.testDppWithoutJoinReorder()'. Dpp will
success even if the fact table and dim table are not adjacent.
* Test the case that the dim side have one dim table but the dim side is an
subQuery: we can write the query like the test
'DynamicPartitionPruningProgramTest.testDppWithSubQuery()'. Dpp will success
even if the dim side is a sub query (Notes: the dim side only can contain one
same table with filter conddiition).
* Test the case that there are 'Union' in fact side: we can write the query
like the test
'DynamicPartitionPruningProgramTest.testDppWithUnionInFactSide()'. Dpp will
success.
* Test the case that there are 'Agg' in fact side: we can write query as the
tests in 'DynamicPartitionPruningProgramTest'. For this case, in order to
ensure the semantic accuracy, we must require that the partition key cannot be
in the group key of 'agg' operator. This case will not success.
3. execute the above plan and verify the execution result. (the execution
result should be same with the execution plan which disable dynamic filtering
via setting 'table.optimizer.dynamic-filtering.enabled = false'
[1] [https://issues.apache.org/jira/browse/FLINK-28940|http://example.com/]
was:
This issue aims to verify the FLINK-31062: New dynamic partition pruning
strategy to support more dpp patterns . This is an extension work of FLIP-248 :
[https://cwiki.apache.org/confluence/display/FLINK/FLIP-248%3A+Introduce+dynamic+partition+pruning]
to enable more query patterns support dynamic partition pruning, like
subQuery, Union etc.
This work can be verified in SQL client after we build the flink-disk package.
# create a partition table and a non-partition table (only hive connector is
supported now), and then insert some data.
# you can verify this work by checking the query plan by executing 'explain'.
Most basic dpp patterns have been verified in Flink-1.16 [1]. So we only need
to focus on verifying the newly supported patterns:
* Test the case that the dim table and the fact table are not adjacent in the
join order: disable join reorder by setting config:
'table.optimizer.join-reorder-enabled = false', and write a query like the
test 'DynamicPartitionPruningProgramTest.testDppWithoutJoinReorder()'. Dpp will
success even if the fact table and dim table are not adjacent.
* Test the case that the dim side have one dim table but the dim side is an
subQuery: we can write the query like the test
'DynamicPartitionPruningProgramTest.testDppWithSuQuery()'. Dpp will success
even if the dim side is a sub query (Notes: the dim side only can contain one
same table with filter conddiition).
* Test the case that there are 'Union' in fact side: we can write the query
like the test
'DynamicPartitionPruningProgramTest.testDppWithUnionInFactSide()'. Dpp will
success.
* Test the case that there are 'Agg' in fact side: we can write query as the
tests in 'DynamicPartitionPruningProgramTest'. For this case, in order to
ensure the semantic accuracy, we must require that the partition key cannot be
in the group key of 'agg' operator. This case will not success.
3. execute the above plan and verify the execution result. (the execution
result should be same with the execution plan which disable dynamic filtering
via setting 'table.optimizer.dynamic-filtering.enabled = false'
[1] [https://issues.apache.org/jira/browse/FLINK-28940|http://example.com/]
> Release Testing: Verify FLINK-30365 New dynamic partition pruning strategy to
> support more dpp patterns
> -------------------------------------------------------------------------------------------------------
>
> Key: FLINK-31062
> URL: https://issues.apache.org/jira/browse/FLINK-31062
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
> Affects Versions: 1.17.0
> Reporter: Yunhong Zheng
> Assignee: Weijie Guo
> Priority: Major
> Fix For: 1.17.0
>
>
> This issue aims to verify the FLINK-31062: New dynamic partition pruning
> strategy to support more dpp patterns . This is an extension work of FLIP-248
> :
> [https://cwiki.apache.org/confluence/display/FLINK/FLIP-248%3A+Introduce+dynamic+partition+pruning]
> to enable more query patterns support dynamic partition pruning, like
> subQuery, Union etc.
> This work can be verified in SQL client after we build the flink-disk package.
> # create a partition table and a non-partition table (only hive connector is
> supported now), and then insert some data.
> # you can verify this work by checking the query plan by executing
> 'explain'. Most basic dpp patterns have been verified in Flink-1.16 [1]. So
> we only need to focus on verifying the newly supported patterns:
> * Test the case that the dim table and the fact table are not adjacent in
> the join order: disable join reorder by setting config:
> 'table.optimizer.join-reorder-enabled = false', and write a query like the
> test 'DynamicPartitionPruningProgramTest.testDppWithoutJoinReorder()'. Dpp
> will success even if the fact table and dim table are not adjacent.
> * Test the case that the dim side have one dim table but the dim side is an
> subQuery: we can write the query like the test
> 'DynamicPartitionPruningProgramTest.testDppWithSubQuery()'. Dpp will success
> even if the dim side is a sub query (Notes: the dim side only can contain one
> same table with filter conddiition).
> * Test the case that there are 'Union' in fact side: we can write the query
> like the test
> 'DynamicPartitionPruningProgramTest.testDppWithUnionInFactSide()'. Dpp will
> success.
> * Test the case that there are 'Agg' in fact side: we can write query as the
> tests in 'DynamicPartitionPruningProgramTest'. For this case, in order to
> ensure the semantic accuracy, we must require that the partition key cannot
> be in the group key of 'agg' operator. This case will not success.
> 3. execute the above plan and verify the execution result. (the
> execution result should be same with the execution plan which disable dynamic
> filtering via setting 'table.optimizer.dynamic-filtering.enabled = false'
>
> [1] [https://issues.apache.org/jira/browse/FLINK-28940|http://example.com/]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
