pvary commented on code in PR #17938:
URL: https://github.com/apache/iceberg/pull/17938#discussion_r3932258072
##########
docs/docs/flink-configuration.md:
##########
@@ -201,3 +207,26 @@ they are.
This is only applicable to [`StatisticsType.Map`](../../javadoc/{{
icebergVersion
}}/org/apache/iceberg/flink/sink/shuffle/StatisticsType.html#Map) for
low-cardinality scenario. For [`StatisticsType.Sketch`](../../javadoc/{{
icebergVersion
}}/org/apache/iceberg/flink/sink/shuffle/StatisticsType.html#Sketch)
high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be
generated during
write. Sketch range partitioner simply splits high-cardinality keys into
ordered ranges.
+
+### Execution options
+
+Iceberg also provides a set of `table.exec.iceberg.*` options that are read
from the Flink
+configuration instead of per-job read or write options. In SQL they are set on
the session:
+
+```sql
+SET 'table.exec.iceberg.infer-source-parallelism' = 'false';
+```
+
+When using the DataStream API, they can be set in the `Configuration` passed
to the source or sink
+builder.
+
+| Flink configuration | Default
| Description
|
+|--------------------------------------------------|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true
| If true, the source parallelism for batch reads is inferred from
the number of scan splits, capped by
`table.exec.iceberg.infer-source-parallelism.max` and by the query limit if one
is set. If false, the source parallelism is taken from the Flink configuration.
Streaming reads never infer parallelism. |
+| table.exec.iceberg.infer-source-parallelism.max | 100
| Maximum inferred parallelism for the source operator.
|
+| table.exec.iceberg.expose-split-locality-info | none
| Whether to expose split host information to use Flink's
locality-aware split assigner. Only applies to the deprecated `FlinkSource`,
i.e. when `table.exec.iceberg.use-flip27-source` is `false`; the FLIP-27
`IcebergSource` ignores this option. If unset, locality information is exposed
automatically when the table's storage can provide block locations (for example
HDFS). |
+| table.exec.iceberg.fetch-batch-record-count | 2048
| Target number of records per fetch batch in the Iceberg source
reader.
|
+| table.exec.iceberg.worker-pool-size | max(2, available cpu)
| Size of the worker pool used to plan or scan manifests. Defaults to the
shared Iceberg worker pool size, which is controlled by the
`iceberg.worker.num-threads` system property.
|
+| table.exec.iceberg.split-assigner-type | simple
| Split assigner type that determines how splits are assigned to
readers. Currently the only type is `simple`, which doesn't provide any
guarantee on order or locality.
|
+| table.exec.iceberg.use-flip27-source | true
| Use the
[FLIP-27](https://cwiki.apache.org/confluence/spaces/FLINK/pages/95653746/FLIP-27+Refactor+Source+Interface)
based `IcebergSource` implementation. Set to false to fall back to the
deprecated `FlinkSource`.
|
Review Comment:
We should start thinking about removing the FlinkSource, so not expose it
now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
