pvary commented on code in PR #7571: URL: https://github.com/apache/iceberg/pull/7571#discussion_r1190871072
##########
docs/flink-configuration.md:
##########
@@ -110,27 +110,28 @@ env.getConfig()
`Read option` has the highest priority, followed by `Flink configuration` and
then `Table property`.
-| Read option | Flink configuration
| Table property | Default | Description
|
-| --------------------------- | ---------------------------------------------
| ---------------------------- | -------------------------------- |
------------------------------------------------------------ |
-| snapshot-id | N/A
| N/A | null | For time
travel in batch mode. Read data from the specified snapshot-id. |
-| case-sensitive | connector.iceberg.case-sensitive
| N/A | false | If true,
match column name in a case sensitive way. |
-| as-of-timestamp | N/A
| N/A | null | For time
travel in batch mode. Read data from the most recent snapshot as of the given
time in milliseconds. |
+| Read option | Flink configuration
| Table property | Default | Description
|
+|-----------------------------|-----------------------------------------------|------------------------------|----------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------|
+| snapshot-id | N/A
| N/A | null | For time
travel in batch mode. Read data from the specified snapshot-id.
|
+| case-sensitive | connector.iceberg.case-sensitive
| N/A | false | If true,
match column name in a case sensitive way.
|
+| as-of-timestamp | N/A
| N/A | null | For time
travel in batch mode. Read data from the most recent snapshot as of the given
time in milliseconds.
|
| starting-strategy | connector.iceberg.starting-strategy
| N/A | INCREMENTAL_FROM_LATEST_SNAPSHOT | Starting
strategy for streaming execution. TABLE_SCAN_THEN_INCREMENTAL: Do a regular
table scan then switch to the incremental mode. The incremental mode starts
from the current snapshot exclusive. INCREMENTAL_FROM_LATEST_SNAPSHOT: Start
incremental mode from the latest snapshot inclusive. If it is an empty map, all
future append snapshots should be discovered.
INCREMENTAL_FROM_EARLIEST_SNAPSHOT: Start incremental mode from the earliest
snapshot inclusive. If it is an empty map, all future append snapshots should
be discovered. INCREMENTAL_FROM_SNAPSHOT_ID: Start incremental mode from a
snapshot with a specific id inclusive. INCREMENTAL_FROM_SNAPSHOT_TIMESTAMP:
Start incremental mode from a snapshot with a specific timestamp inclusive. If
the timestamp is between two snapshots, it should start from the snapshot after
the timestamp. Just fo
r FIP27 Source. |
-| start-snapshot-timestamp | N/A
| N/A | null | Start to
read data from the most recent snapshot as of the given time in milliseconds. |
-| start-snapshot-id | N/A
| N/A | null | Start to
read data from the specified snapshot-id. |
-| end-snapshot-id | N/A
| N/A | The latest snapshot id | Specifies
the end snapshot.
-| branch | N/A
| N/A | main | Specifies the branch to read from in batch mode
-| tag | N/A
| N/A | null | Specifies the tag to read from in batch mode
-| start-tag | N/A
| N/A | null | Specifies the starting tag to read from for
incremental reads
-| end-tag | N/A
| N/A | null | Specifies the ending tag to to read from for
incremental reads |
-| split-size | connector.iceberg.split-size
| read.split.target-size | 128 MB | Target size
when combining input splits. |
-| split-lookback | connector.iceberg.split-file-open-cost
| read.split.planning-lookback | 10 | Number of
bins to consider when combining input splits. |
-| split-file-open-cost | connector.iceberg.split-file-open-cost
| read.split.open-file-cost | 4MB | The
estimated cost to open a file, used as a minimum weight when combining splits. |
-| streaming | connector.iceberg.streaming
| N/A | false | Sets
whether the current task runs in streaming or batch mode. |
-| monitor-interval | connector.iceberg.monitor-interval
| N/A | 60s | Monitor
interval to discover splits from new snapshots. Applicable only for streaming
read. |
-| include-column-stats | connector.iceberg.include-column-stats
| N/A | false | Create a
new scan from this that loads the column stats with each data file. Column
stats include: value count, null value count, lower bounds, and upper bounds. |
-| max-planning-snapshot-count | connector.iceberg.max-planning-snapshot-count
| N/A | Integer.MAX_VALUE | Max number
of snapshots limited per split enumeration. Applicable only to streaming read. |
-| limit | connector.iceberg.limit
| N/A | -1 | Limited
output number of rows. |
+| start-snapshot-timestamp | N/A
| N/A | null | Start to
read data from the most recent snapshot as of the given time in milliseconds.
|
+| start-snapshot-id | N/A
| N/A | null | Start to
read data from the specified snapshot-id.
|
+| end-snapshot-id | N/A
| N/A | The latest snapshot id | Specifies
the end snapshot.
|
+| branch | N/A
| N/A | main | Specifies
the branch to read from in batch mode
|
+| tag | N/A
| N/A | null | Specifies
the tag to read from in batch mode
|
+| start-tag | N/A
| N/A | null | Specifies
the starting tag to read from for incremental reads
|
+| end-tag | N/A
| N/A | null | Specifies
the ending tag to to read from for incremental reads
|
+| split-size | connector.iceberg.split-size
| read.split.target-size | 128 MB | Target size
when combining input splits.
|
+| split-lookback | connector.iceberg.split-file-open-cost
| read.split.planning-lookback | 10 | Number of
bins to consider when combining input splits.
|
+| split-file-open-cost | connector.iceberg.split-file-open-cost
| read.split.open-file-cost | 4MB | The
estimated cost to open a file, used as a minimum weight when combining splits.
|
+| streaming | connector.iceberg.streaming
| N/A | false | Sets
whether the current task runs in streaming or batch mode.
|
+| monitor-interval | connector.iceberg.monitor-interval
| N/A | 60s | Monitor
interval to discover splits from new snapshots. Applicable only for streaming
read.
|
+| include-column-stats | connector.iceberg.include-column-stats
| N/A | false | Create a
new scan from this that loads the column stats with each data file. Column
stats include: value count, null value count, lower bounds, and upper bounds.
|
+| max-planning-snapshot-count | connector.iceberg.max-planning-snapshot-count
| N/A | Integer.MAX_VALUE | Max number
of snapshots limited per split enumeration. Applicable only to streaming read.
|
+| limit | connector.iceberg.limit
| N/A | -1 | Limited
output number of rows.
|
+| plan-retry-num | connector.iceberg.plan-retry-num
| N/A | 3 | Retries
before job failure when there is an error during continuous planning. Set to -1
to not fail on error.
|
Review Comment:
After some back and forth, I have decided to go ahead with the
`max-allowed-planning-failures`.
I agree with you that your name is better, but decided to keep the
continuous only in the description to decrease the config and the variable name
lengths
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkReadOptions.java:
##########
@@ -105,4 +105,8 @@ private FlinkReadOptions() {}
public static final String LIMIT = "limit";
public static final ConfigOption<Long> LIMIT_OPTION =
ConfigOptions.key(PREFIX + LIMIT).longType().defaultValue(-1L);
+
+ public static final String PLAN_RETRY_NUM = "plan-retry-num";
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
