pvary commented on code in PR #17938:
URL: https://github.com/apache/iceberg/pull/17938#discussion_r3932094897
##########
docs/docs/flink-configuration.md:
##########
@@ -201,3 +207,26 @@ they are.
This is only applicable to [`StatisticsType.Map`](../../javadoc/{{
icebergVersion
}}/org/apache/iceberg/flink/sink/shuffle/StatisticsType.html#Map) for
low-cardinality scenario. For [`StatisticsType.Sketch`](../../javadoc/{{
icebergVersion
}}/org/apache/iceberg/flink/sink/shuffle/StatisticsType.html#Sketch)
high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be
generated during
write. Sketch range partitioner simply splits high-cardinality keys into
ordered ranges.
+
+### Execution options
+
+Iceberg also provides a set of `table.exec.iceberg.*` options that are read
from the Flink
+configuration instead of per-job read or write options. In SQL they are set on
the session:
+
+```sql
+SET 'table.exec.iceberg.infer-source-parallelism' = 'false';
+```
+
+When using the DataStream API, they can be set in the `Configuration` passed
to the source or sink
+builder.
+
+| Flink configuration | Default
| Description
|
+|--------------------------------------------------|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true
| If true, the source parallelism for batch reads is inferred from
the number of scan splits, capped by
`table.exec.iceberg.infer-source-parallelism.max` and by the query limit if one
is set. If false, the source parallelism is taken from the Flink configuration.
Streaming reads never infer parallelism. |
+| table.exec.iceberg.infer-source-parallelism.max | 100
| Maximum inferred parallelism for the source operator.
|
+| table.exec.iceberg.expose-split-locality-info | none
| Whether to expose split host information to use Flink's
locality-aware split assigner. Only applies to the deprecated `FlinkSource`,
i.e. when `table.exec.iceberg.use-flip27-source` is `false`; the FLIP-27
`IcebergSource` ignores this option. If unset, locality information is exposed
automatically when the table's storage can provide block locations (for example
HDFS). |
Review Comment:
I'm not sure who uses this feature. I think we should deprecate it and
remove instead of exposing it in the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
