2162174202-sudo opened a new issue, #19051:
URL: https://github.com/apache/iceberg/issues/19051
### Apache Iceberg version
1.11.0
### Query engine
Flink
### Please describe the bug 🐞
In Flink batch mode, planning (`EXPLAIN PLAN FOR`) a query that embeds a
non-correlated scalar subquery containing a `LEFT JOIN` (with a derived table)
and a `RIGHT JOIN` inside an outer Iceberg table scan consistently times out
(>60s) on a 4-row Iceberg table.
This reproduces on both **Flink 2.1.0 + Iceberg 1.11.0** (newest supported
pairing) and **Flink 1.19.1 + Iceberg 1.8.1**. By contrast, the exact same SQL
shape and 4-row dataset over Flink's built-in `filesystem` CSV connector plans
in under 1 second on Flink 2.3.0, indicating a planning interaction specific to
the Iceberg table source rather than a generic Flink SQL-shape issue.
### Environment
- **Primary reproduction**:
- Image: `flink:2.1.0-scala_2.12-java17`
- Iceberg runtime: `iceberg-flink-runtime-2.1-1.11.0.jar`
- FileIO: `org.apache.iceberg.aws.s3.S3FileIO`
(`iceberg-aws-bundle-1.11.0.jar`)
- Catalog: Iceberg REST catalog backed by MinIO/S3
- Execution mode: `SET 'execution.runtime-mode' = 'batch';`
- **Earlier reproduction**:
- Flink `1.19.1` + Iceberg `1.8.1`
### Steps to reproduce
1. Configure Flink SQL Gateway in batch mode with an Iceberg REST catalog
and a 4-row Iceberg table (`order_id INT, region STRING, v INT`):
```sql
SET 'execution.runtime-mode' = 'batch';
SET 'table.dynamic-table-options.enabled' = 'true';
CREATE CATALOG iceberg WITH (
'type' = 'iceberg',
'catalog-type' = 'rest',
'uri' = 'http://iceberg-rest:8181',
'warehouse' = 's3://warehouse/wh/',
'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO',
's3.endpoint' = 'http://minio:9000',
's3.path-style-access' = 'true',
's3.access-key-id' = 'admin',
's3.secret-access-key' = 'password'
);
USE CATALOG iceberg;
-- Table schema and 4-row fixture used in our tests:
-- CREATE TABLE ld_pd_target_20261006a (order_id INT, region STRING, v INT);
-- INSERT INTO ld_pd_target_20261006a VALUES (1, 'west', 11), (2, 'west',
12), (5, 'east', 15), (6, 'east', 16);
```
2. Run `EXPLAIN PLAN FOR` (in our reproduction, the 4-row snapshot was
pinned with `/*+ OPTIONS('snapshot-id'='2218517376986569101') */`; when
reproducing on a freshly created table, you can omit the `snapshot-id` hint):
```sql
EXPLAIN PLAN FOR
SELECT
((SELECT t4.order_id
FROM ld_pd_target_20261006a /*+
OPTIONS('snapshot-id'='2218517376986569101') */ AS t2
LEFT JOIN (
SELECT x.region AS q2_col_1, x.v AS q2_col_2, x.v AS q2_col_3
FROM ld_pd_target_20261006a /*+
OPTIONS('snapshot-id'='2218517376986569101') */ AS x
) AS sq3 ON t2.region <> sq3.q2_col_1
RIGHT JOIN ld_pd_target_20261006a /*+
OPTIONS('snapshot-id'='2218517376986569101') */ AS t4
ON sq3.q2_col_1 NOT LIKE '%sample_19%'
ORDER BY t4.order_id DESC
LIMIT 1) < t1.v AND t1.order_id BETWEEN 29 AND 65),
t1.order_id,
t1.v
FROM ld_pd_target_20261006a /*+ OPTIONS('snapshot-id'='2218517376986569101')
*/ AS t1;
```
### Observed behavior
**Flink 2.1.0 + Iceberg 1.11.0 (60s operation timeout, `EXPLAIN PLAN FOR`
only):**
| Case | Status | Elapsed |
|---|---|---:|
| `EXPLAIN PLAN FOR SELECT 1` | ok | 11,254 ms |
| Control Iceberg scan (`SELECT order_id, region, v FROM
ld_pd_target_20261006a LIMIT 1`) | ok | 6,430 ms |
| Reproducer attempt 1 | `TimeoutError` (>60s) | 65,099 ms |
| Reproducer attempt 2 | `TimeoutError` (>60s) | 66,683 ms |
| Reproducer attempt 3 | `TimeoutError` (>60s) | 67,511 ms |
**Flink 1.19.1 + Iceberg 1.8.1 (20s operation timeout, 3 independent healthy
sessions):**
- Reproducer timed out 3/3 times (`25,184 ms`, `25,127 ms`, `25,073 ms`
including timeout cleanup), while health checks before each attempt succeeded
in `2,105 ms`, `1,913 ms`, and `1,469 ms`.
### Isolation & controls
- **Flink 2.3.0 filesystem CSV control (same 4 rows & identical SQL shape,
`EXPLAIN PLAN FOR`)**: succeeded 3/3 in `939 ms`, `759 ms`, and `489 ms`
(identical 5,126-character plan).
- **Standalone scalar subquery** (without outer Iceberg scan): planned in
`1,653 ms` and executed in `2,575 ms` on Flink 1.19.1.
- **Equivalent pure `VALUES` query**: executed in `1,855 ms`.
- **Disabling source reuse or sub-plan reuse**: did not remove the timeout.
### Expected behavior
`EXPLAIN PLAN FOR` on a 4-row Iceberg table should complete planning
promptly (comparable to the control scan and filesystem connector) rather than
hanging until timeout.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]