[
https://issues.apache.org/jira/browse/FLINK-37971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels updated FLINK-37971:
---------------------------------------
Description:
The IcebergSource supports using SQL hints to set some runtime properties like
whether to run in batch or streaming mode. For example, this creates an Iceberg
source and an Iceberg sink:
{noformat}
INSERT into output
SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/
;
{noformat}
In the below query, the resulting DAG would contain only one IcebergSource and
two Iceberg sinks:
{noformat}
INSERT into output1
SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/
;
INSERT INTO output2
SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/
;
{noformat}
However, the SQL hints will not be available when creating the source, i.e. the
{{FlinkDynamicTableFactory#createDynamicTableSource(Context context}} will not
receive the options derived from the SQL hint via
{{context.getCatalogTable().getOptions()}}, as is the case before the optimizer
combines the two SELECT statements.
It starts to work again when {{table.optimizer.reuse-source-enabled: false}} is
set.
was:
The IcebergSource supports using SQL hints to set some runtime properties like
whether to run in batch or streaming mode. For example, this creates an Iceberg
source and an Iceberg sink:
{noformat}
INSERT into output
SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/
;
{noformat}
In the below query, the resulting DAG would contain only one IcebergSource and
two Iceberg sinks:
{noformat}
INSERT into output1
SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/
;
INSERT INTO output2
SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/
;
{noformat}
However, the SQL hints will not be applied at the source, i.e. the
{{DynamicTableSourceFactory}} and the resulting {{ResolvedCatalogTable}} will
not contain the options derived from the SQL hint.
> SQL hints are dropped when combining multiple SELECT into a single source
> -------------------------------------------------------------------------
>
> Key: FLINK-37971
> URL: https://issues.apache.org/jira/browse/FLINK-37971
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Table SQL / Runtime
> Affects Versions: 2.0.0, 1.19.2, 1.20.1
> Reporter: Maximilian Michels
> Priority: Major
>
> The IcebergSource supports using SQL hints to set some runtime properties
> like whether to run in batch or streaming mode. For example, this creates an
> Iceberg source and an Iceberg sink:
> {noformat}
> INSERT into output
> SELECT * FROM sample /*+ OPTIONS('streaming'='true',
> 'monitor-interval'='1s')*/ ;
> {noformat}
> In the below query, the resulting DAG would contain only one IcebergSource
> and two Iceberg sinks:
> {noformat}
> INSERT into output1
> SELECT * FROM sample /*+ OPTIONS('streaming'='true',
> 'monitor-interval'='1s')*/ ;
> INSERT INTO output2
> SELECT * FROM sample /*+ OPTIONS('streaming'='true',
> 'monitor-interval'='1s')*/ ;
> {noformat}
> However, the SQL hints will not be available when creating the source, i.e.
> the {{FlinkDynamicTableFactory#createDynamicTableSource(Context context}}
> will not receive the options derived from the SQL hint via
> {{context.getCatalogTable().getOptions()}}, as is the case before the
> optimizer combines the two SELECT statements.
> It starts to work again when {{table.optimizer.reuse-source-enabled: false}}
> is set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
