stevenzwu commented on a change in pull request #4060:
URL: https://github.com/apache/iceberg/pull/4060#discussion_r802097100
##########
File path:
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java
##########
@@ -40,4 +40,10 @@ private FlinkConfigOptions() {
.intType()
.defaultValue(100)
.withDescription("Sets max infer parallelism for source operator.");
+
+ public static final ConfigOption<Integer>
SOURCE_READER_FETCH_BATCH_RECORD_COUNT = ConfigOptions
+ .key("source.iceberg.reader.fetch-batch-record-count")
Review comment:
This is a config/property for Iceberg connector.
User can set it programmatically when constructing the FLIP-27
`IcebergSource`. Below is a code snippet from unit test.
```
Configuration config = new Configuration();
config.setInteger(FlinkConfigOptions.SOURCE_READER_FETCH_BATCH_RECORD_COUNT,
128);
IcebergSource.Builder<RowData> sourceBuilder =
IcebergSource.<RowData>builder()
.tableLoader(tableLoader())
.assignerFactory(new SimpleSplitAssignerFactory())
.readerFunction(new RowDataReaderFunction(config, table,
projectedSchema, null, false));
```
On the table/SQL side, Flink Kafka connector has separate table connector
config. E.g., `KafkaConnectorOptions.SCAN_TOPIC_PARTITION_DISCOVERY` is defined
as `scan.topic-partition-discovery.interval` as documented here.
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/#how-to-create-a-kafka-table
Flink `KafkaDynamicTableFactory` or Iceberg `FlinkDynamicTableFactory` can
extract above table connector config when creating `DynamicTableSource`.
With that said, maybe we should name the config as
`reader.fetch.batch-record-count` without the `source.iceberg` prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
