twalthr commented on a change in pull request #18290:
URL: https://github.com/apache/flink/pull/18290#discussion_r784049657
##########
File path: docs/content/docs/connectors/table/kinesis.md
##########
@@ -255,349 +271,399 @@ Connector Options
<tr>
<td><h5>scan.stream.initpos</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">LATEST</td>
<td>String</td>
<td>Initial position to be used when reading from the table. See <a
href="#start-reading-position">Start Reading Position</a> for details.</td>
</tr>
<tr>
<td><h5>scan.stream.initpos-timestamp</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>The initial timestamp to start reading Kinesis stream from (when
<code>scan.stream.initpos</code> is AT_TIMESTAMP). See <a
href="#start-reading-position">Start Reading Position</a> for details.</td>
</tr>
<tr>
<td><h5>scan.stream.initpos-timestamp-format</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">yyyy-MM-dd'T'HH:mm:ss.SSSXXX</td>
<td>String</td>
<td>The date format of initial timestamp to start reading Kinesis stream
from (when <code>scan.stream.initpos</code> is AT_TIMESTAMP). See <a
href="#start-reading-position">Start Reading Position</a> for details.</td>
</tr>
<tr>
<td><h5>scan.stream.recordpublisher</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">POLLING</td>
<td>String</td>
<td>The <code>RecordPublisher</code> type to use for sources. See <a
href="#enhanced-fan-out">Enhanced Fan-Out</a> for details.</td>
</tr>
<tr>
<td><h5>scan.stream.efo.consumername</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>The name of the EFO consumer to register with KDS. See <a
href="#enhanced-fan-out">Enhanced Fan-Out</a> for details.</td>
</tr>
<tr>
<td><h5>scan.stream.efo.registration</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">LAZY</td>
<td>String</td>
<td>Determine how and when consumer de-/registration is performed
(LAZY|EAGER|NONE). See <a href="#enhanced-fan-out">Enhanced Fan-Out</a> for
details.</td>
</tr>
<tr>
<td><h5>scan.stream.efo.consumerarn</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>The prefix of consumer ARN for a given stream. See <a
href="#enhanced-fan-out">Enhanced Fan-Out</a> for details.</td>
</tr>
<tr>
<td><h5>scan.stream.efo.http-client.max-concurrency</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">10000</td>
<td>Integer</td>
<td>Maximum number of allowed concurrent requests for the EFO client.
See <a href="#enhanced-fan-out">Enhanced Fan-Out</a> for details.</td>
</tr>
<tr>
<td><h5>scan.stream.describe.maxretries</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">50</td>
<td>Integer</td>
<td>The maximum number of <code>describeStream</code> attempts if we get
a recoverable exception.</td>
</tr>
<tr>
<td><h5>scan.stream.describe.backoff.base</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">2000</td>
<td>Long</td>
<td>The base backoff time (in milliseconds) between each
<code>describeStream</code> attempt (for consuming from DynamoDB streams).</td>
</tr>
<tr>
<td><h5>scan.stream.describe.backoff.max</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">5000</td>
<td>Long</td>
<td>The maximum backoff time (in milliseconds) between each
<code>describeStream</code> attempt (for consuming from DynamoDB streams).</td>
</tr>
<tr>
<td><h5>scan.stream.describe.backoff.expconst</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1.5</td>
<td>Double</td>
<td>The power constant for exponential backoff between each
<code>describeStream</code> attempt (for consuming from DynamoDB streams).</td>
</tr>
<tr>
<td><h5>scan.list.shards.maxretries</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">10</td>
<td>Integer</td>
<td>The maximum number of <code>listShards</code> attempts if we get a
recoverable exception.</td>
</tr>
<tr>
<td><h5>scan.list.shards.backoff.base</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1000</td>
<td>Long</td>
<td>The base backoff time (in milliseconds) between each
<code>listShards</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.list.shards.backoff.max</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">5000</td>
<td>Long</td>
<td>The maximum backoff time (in milliseconds) between each
<code>listShards</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.list.shards.backoff.expconst</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1.5</td>
<td>Double</td>
<td>The power constant for exponential backoff between each
<code>listShards</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.stream.describestreamconsumer.maxretries</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">50</td>
<td>Integer</td>
<td>The maximum number of <code>describeStreamConsumer</code> attempts
if we get a recoverable exception.</td>
</tr>
<tr>
<td><h5>scan.stream.describestreamconsumer.backoff.base</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">2000</td>
<td>Long</td>
<td>The base backoff time (in milliseconds) between each
<code>describeStreamConsumer</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.stream.describestreamconsumer.backoff.max</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">5000</td>
<td>Long</td>
<td>The maximum backoff time (in milliseconds) between each
<code>describeStreamConsumer</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.stream.describestreamconsumer.backoff.expconst</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1.5</td>
<td>Double</td>
<td>The power constant for exponential backoff between each
<code>describeStreamConsumer</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.stream.registerstreamconsumer.maxretries</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">10</td>
<td>Integer</td>
<td>The maximum number of <code>registerStream</code> attempts if we get
a recoverable exception.</td>
</tr>
<tr>
<td><h5>scan.stream.registerstreamconsumer.timeout</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">60</td>
<td>Integer</td>
<td>The maximum time in seconds to wait for a stream consumer to become
active before giving up.</td>
</tr>
<tr>
<td><h5>scan.stream.registerstreamconsumer.backoff.base</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">500</td>
<td>Long</td>
<td>The base backoff time (in milliseconds) between each
<code>registerStream</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.stream.registerstreamconsumer.backoff.max</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">2000</td>
<td>Long</td>
<td>The maximum backoff time (in milliseconds) between each
<code>registerStream</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.stream.registerstreamconsumer.backoff.expconst</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1.5</td>
<td>Double</td>
<td>The power constant for exponential backoff between each
<code>registerStream</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.stream.deregisterstreamconsumer.maxretries</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">10</td>
<td>Integer</td>
<td>The maximum number of <code>deregisterStream</code> attempts if we
get a recoverable exception.</td>
</tr>
<tr>
<td><h5>scan.stream.deregisterstreamconsumer.timeout</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">60</td>
<td>Integer</td>
<td>The maximum time in seconds to wait for a stream consumer to
deregister before giving up.</td>
</tr>
<tr>
<td><h5>scan.stream.deregisterstreamconsumer.backoff.base</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">500</td>
<td>Long</td>
<td>The base backoff time (in milliseconds) between each
<code>deregisterStream</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.stream.deregisterstreamconsumer.backoff.max</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">2000</td>
<td>Long</td>
<td>The maximum backoff time (in milliseconds) between each
<code>deregisterStream</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.stream.deregisterstreamconsumer.backoff.expconst</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1.5</td>
<td>Double</td>
<td>The power constant for exponential backoff between each
<code>deregisterStream</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.shard.subscribetoshard.maxretries</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">10</td>
<td>Integer</td>
<td>The maximum number of <code>subscribeToShard</code> attempts if we
get a recoverable exception.</td>
</tr>
<tr>
<td><h5>scan.shard.subscribetoshard.backoff.base</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1000</td>
<td>Long</td>
<td>The base backoff time (in milliseconds) between each
<code>subscribeToShard</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.shard.subscribetoshard.backoff.max</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">2000</td>
<td>Long</td>
<td>The maximum backoff time (in milliseconds) between each
<code>subscribeToShard</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.shard.subscribetoshard.backoff.expconst</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1.5</td>
<td>Double</td>
<td>The power constant for exponential backoff between each
<code>subscribeToShard</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.shard.getrecords.maxrecordcount</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">10000</td>
<td>Integer</td>
<td>The maximum number of records to try to get each time we fetch
records from a AWS Kinesis shard.</td>
</tr>
<tr>
<td><h5>scan.shard.getrecords.maxretries</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">3</td>
<td>Integer</td>
<td>The maximum number of <code>getRecords</code> attempts if we get a
recoverable exception.</td>
</tr>
<tr>
<td><h5>scan.shard.getrecords.backoff.base</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">300</td>
<td>Long</td>
<td>The base backoff time (in milliseconds) between
<code>getRecords</code> attempts if we get a
ProvisionedThroughputExceededException.</td>
</tr>
<tr>
<td><h5>scan.shard.getrecords.backoff.max</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1000</td>
<td>Long</td>
<td>The maximum backoff time (in milliseconds) between
<code>getRecords</code> attempts if we get a
ProvisionedThroughputExceededException.</td>
</tr>
<tr>
<td><h5>scan.shard.getrecords.backoff.expconst</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1.5</td>
<td>Double</td>
<td>The power constant for exponential backoff between each
<code>getRecords</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.shard.getrecords.intervalmillis</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">200</td>
<td>Long</td>
<td>The interval (in milliseconds) between each <code>getRecords</code>
request to a AWS Kinesis shard in milliseconds.</td>
</tr>
<tr>
<td><h5>scan.shard.getiterator.maxretries</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">3</td>
<td>Integer</td>
<td>The maximum number of <code>getShardIterator</code> attempts if we
get ProvisionedThroughputExceededException.</td>
</tr>
<tr>
<td><h5>scan.shard.getiterator.backoff.base</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">300</td>
<td>Long</td>
<td>The base backoff time (in milliseconds) between
<code>getShardIterator</code> attempts if we get a
ProvisionedThroughputExceededException.</td>
</tr>
<tr>
<td><h5>scan.shard.getiterator.backoff.max</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1000</td>
<td>Long</td>
<td>The maximum backoff time (in milliseconds) between
<code>getShardIterator</code> attempts if we get a
ProvisionedThroughputExceededException.</td>
</tr>
<tr>
<td><h5>scan.shard.getiterator.backoff.expconst</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">1.5</td>
<td>Double</td>
<td>The power constant for exponential backoff between each
<code>getShardIterator</code> attempt.</td>
</tr>
<tr>
<td><h5>scan.shard.discovery.intervalmillis</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">10000</td>
<td>Integer</td>
<td>The interval between each attempt to discover new shards.</td>
</tr>
<tr>
<td><h5>scan.shard.adaptivereads</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">false</td>
<td>Boolean</td>
<td>The config to turn on adaptive reads from a shard. See the
<code>AdaptivePollingRecordPublisher</code> documentation for details.</td>
</tr>
<tr>
<td><h5>scan.shard.idle.interval</h5></td>
<td>optional</td>
+ <td>no</td>
<td style="word-wrap: break-word;">-1</td>
<td>Long</td>
<td>The interval (in milliseconds) after which to consider a shard idle
for purposes of watermark generation. A positive value will allow the watermark
to progress even when some shards don't receive new records.</td>
</tr>
<tr>
<td><h5>scan.watermark.sync.interval</h5></td>
<td>optional</td>
+ <td>no</td>
Review comment:
With the new dual representation of `mapType` I guess we could update
the implementation and make it possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
