This is an automated email from the ASF dual-hosted git repository. guoweijie pushed a commit to branch v3.0.0-docs in repository https://gitbox.apache.org/repos/asf/flink-connector-pulsar.git
commit fb7a44252e111cb081aeef0030f094d702979bc7 Author: Yuxin Tan <[email protected]> AuthorDate: Tue Apr 4 15:30:45 2023 +0800 [FLINK-31720][docs][Connector/Pulsar] Add the pulsar connector docs --- .../generated/pulsar_admin_configuration.html | 42 ++++ .../generated/pulsar_client_configuration.html | 222 +++++++++++++++++++++ .../generated/pulsar_consumer_configuration.html | 168 ++++++++++++++++ .../generated/pulsar_producer_configuration.html | 78 ++++++++ .../generated/pulsar_sink_configuration.html | 54 +++++ .../generated/pulsar_source_configuration.html | 60 ++++++ 6 files changed, 624 insertions(+) diff --git a/docs/layouts/shortcodes/generated/pulsar_admin_configuration.html b/docs/layouts/shortcodes/generated/pulsar_admin_configuration.html new file mode 100644 index 0000000..31c7273 --- /dev/null +++ b/docs/layouts/shortcodes/generated/pulsar_admin_configuration.html @@ -0,0 +1,42 @@ +<table class="configuration table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 20%">Key</th> + <th class="text-left" style="width: 15%">Default</th> + <th class="text-left" style="width: 10%">Type</th> + <th class="text-left" style="width: 55%">Description</th> + </tr> + </thead> + <tbody> + <tr> + <td><h5>pulsar.admin.adminUrl</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>The Pulsar service HTTP URL for the admin endpoint. For example, <code class="highlighter-rouge">http://my-broker.example.com:8080</code>, or <code class="highlighter-rouge">https://my-broker.example.com:8443</code> for TLS.</td> + </tr> + <tr> + <td><h5>pulsar.admin.autoCertRefreshTime</h5></td> + <td style="word-wrap: break-word;">300000</td> + <td>Integer</td> + <td>The auto cert refresh time (in ms) if Pulsar admin supports TLS authentication.</td> + </tr> + <tr> + <td><h5>pulsar.admin.connectTimeout</h5></td> + <td style="word-wrap: break-word;">60000</td> + <td>Integer</td> + <td>The connection time out (in ms) for the PulsarAdmin client.</td> + </tr> + <tr> + <td><h5>pulsar.admin.readTimeout</h5></td> + <td style="word-wrap: break-word;">60000</td> + <td>Integer</td> + <td>The server response read timeout (in ms) for the PulsarAdmin client for any request.</td> + </tr> + <tr> + <td><h5>pulsar.admin.requestTimeout</h5></td> + <td style="word-wrap: break-word;">300000</td> + <td>Integer</td> + <td>The server request timeout (in ms) for the PulsarAdmin client for any request.</td> + </tr> + </tbody> +</table> diff --git a/docs/layouts/shortcodes/generated/pulsar_client_configuration.html b/docs/layouts/shortcodes/generated/pulsar_client_configuration.html new file mode 100644 index 0000000..970b847 --- /dev/null +++ b/docs/layouts/shortcodes/generated/pulsar_client_configuration.html @@ -0,0 +1,222 @@ +<table class="configuration table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 20%">Key</th> + <th class="text-left" style="width: 15%">Default</th> + <th class="text-left" style="width: 10%">Type</th> + <th class="text-left" style="width: 55%">Description</th> + </tr> + </thead> + <tbody> + <tr> + <td><h5>pulsar.client.authParamMap</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>Map</td> + <td>Parameters for the authentication plugin.</td> + </tr> + <tr> + <td><h5>pulsar.client.authParams</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>Parameters for the authentication plugin.<br /><br />Example:<br /><code class="highlighter-rouge">key1:val1,key2:val2</code></td> + </tr> + <tr> + <td><h5>pulsar.client.authPluginClassName</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>Name of the authentication plugin.</td> + </tr> + <tr> + <td><h5>pulsar.client.concurrentLookupRequest</h5></td> + <td style="word-wrap: break-word;">5000</td> + <td>Integer</td> + <td>The number of concurrent lookup requests allowed to send on each broker connection to prevent overload on the broker. It should be configured with a higher value only in case of it requires to produce or subscribe on thousands of topic using a created <code class="highlighter-rouge">PulsarClient</code></td> + </tr> + <tr> + <td><h5>pulsar.client.connectionTimeoutMs</h5></td> + <td style="word-wrap: break-word;">10000</td> + <td>Integer</td> + <td>Duration (in ms) of waiting for a connection to a broker to be established.<br />If the duration passes without a response from a broker, the connection attempt is dropped.</td> + </tr> + <tr> + <td><h5>pulsar.client.connectionsPerBroker</h5></td> + <td style="word-wrap: break-word;">1</td> + <td>Integer</td> + <td>The maximum number of connections that the client library will open to a single broker.<br /> By default, the connection pool will use a single connection for all the producers and consumers. Increasing this parameter may improve throughput when using many producers over a high latency connection.</td> + </tr> + <tr> + <td><h5>pulsar.client.enableBusyWait</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>Option to enable busy-wait settings.<br />This option will enable spin-waiting on executors and IO threads in order to reduce latency during context switches. The spinning will consume 100% CPU even when the broker is not doing any work. It is recommended to reduce the number of IO threads and BookKeeper client threads to only have fewer CPU cores busy.</td> + </tr> + <tr> + <td><h5>pulsar.client.enableTransaction</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>If transaction is enabled, start the <code class="highlighter-rouge">transactionCoordinatorClient</code> with <code class="highlighter-rouge">PulsarClient</code>.</td> + </tr> + <tr> + <td><h5>pulsar.client.initialBackoffIntervalNanos</h5></td> + <td style="word-wrap: break-word;">100000000</td> + <td>Long</td> + <td>Default duration (in nanoseconds) for a backoff interval.</td> + </tr> + <tr> + <td><h5>pulsar.client.keepAliveIntervalSeconds</h5></td> + <td style="word-wrap: break-word;">30</td> + <td>Integer</td> + <td>Interval (in seconds) for keeping connection between the Pulsar client and broker alive.</td> + </tr> + <tr> + <td><h5>pulsar.client.listenerName</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>Configure the <code class="highlighter-rouge">listenerName</code> that the broker will return the corresponding <code class="highlighter-rouge">advertisedListener</code>.</td> + </tr> + <tr> + <td><h5>pulsar.client.maxBackoffIntervalNanos</h5></td> + <td style="word-wrap: break-word;">60000000000</td> + <td>Long</td> + <td>The maximum duration (in nanoseconds) for a backoff interval.</td> + </tr> + <tr> + <td><h5>pulsar.client.maxLookupRedirects</h5></td> + <td style="word-wrap: break-word;">20</td> + <td>Integer</td> + <td>The maximum number of times a lookup-request redirections to a broker.</td> + </tr> + <tr> + <td><h5>pulsar.client.maxLookupRequest</h5></td> + <td style="word-wrap: break-word;">50000</td> + <td>Integer</td> + <td>The maximum number of lookup requests allowed on each broker connection to prevent overload on the broker. It should be greater than <code class="highlighter-rouge">pulsar.client.concurrentLookupRequest</code>. Requests that inside <code class="highlighter-rouge">pulsar.client.concurrentLookupRequest</code> are already sent to broker, and requests beyond <code class="highlighter-rouge">pulsar.client.concurrentLookupRequest</code> and under <code class="highlighter-rouge"> [...] + </tr> + <tr> + <td><h5>pulsar.client.maxNumberOfRejectedRequestPerConnection</h5></td> + <td style="word-wrap: break-word;">50</td> + <td>Integer</td> + <td>The maximum number of rejected requests of a broker in a certain period (30s) after the current connection is closed and the client creates a new connection to connect to a different broker.</td> + </tr> + <tr> + <td><h5>pulsar.client.memoryLimitBytes</h5></td> + <td style="word-wrap: break-word;">67108864</td> + <td>Long</td> + <td>The limit (in bytes) on the amount of direct memory that will be allocated by this client instance.<br />Note: at this moment this is only limiting the memory for producers. Setting this to <code class="highlighter-rouge">0</code> will disable the limit.</td> + </tr> + <tr> + <td><h5>pulsar.client.numIoThreads</h5></td> + <td style="word-wrap: break-word;">1</td> + <td>Integer</td> + <td>The number of threads used for handling connections to brokers.</td> + </tr> + <tr> + <td><h5>pulsar.client.numListenerThreads</h5></td> + <td style="word-wrap: break-word;">1</td> + <td>Integer</td> + <td>The number of threads used for handling message listeners. The listener thread pool is shared across all the consumers and readers that are using a <code class="highlighter-rouge">listener</code> model to get messages. For a given consumer, the listener is always invoked from the same thread to ensure ordering.</td> + </tr> + <tr> + <td><h5>pulsar.client.operationTimeoutMs</h5></td> + <td style="word-wrap: break-word;">30000</td> + <td>Integer</td> + <td>Operation timeout (in ms). Operations such as creating producers, subscribing or unsubscribing topics are retried during this interval. If the operation is not completed during this interval, the operation will be marked as failed.</td> + </tr> + <tr> + <td><h5>pulsar.client.proxyProtocol</h5></td> + <td style="word-wrap: break-word;">SNI</td> + <td><p>Enum</p></td> + <td>Protocol type to determine the type of proxy routing when a client connects to the proxy using <code class="highlighter-rouge">pulsar.client.proxyServiceUrl</code>.<br /><br />Possible values:<ul><li>"SNI"</li></ul></td> + </tr> + <tr> + <td><h5>pulsar.client.proxyServiceUrl</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>Proxy-service URL when a client connects to the broker via the proxy. The client can choose the type of proxy-routing.</td> + </tr> + <tr> + <td><h5>pulsar.client.requestTimeoutMs</h5></td> + <td style="word-wrap: break-word;">60000</td> + <td>Integer</td> + <td>Maximum duration (in ms) for completing a request. This config option is not supported before Pulsar 2.8.1</td> + </tr> + <tr> + <td><h5>pulsar.client.serviceUrl</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>Service URL provider for Pulsar service.<br />To connect to Pulsar using client libraries, you need to specify a Pulsar protocol URL.<br />You can assign Pulsar protocol URLs to specific clusters and use the Pulsar scheme.<br /><ul><li>This is an example of <code class="highlighter-rouge">localhost</code>: <code class="highlighter-rouge">pulsar://localhost:6650</code>.</li><li>If you have multiple brokers, the URL is as: <code class="highlighter-rouge">pulsar://localhost: [...] + </tr> + <tr> + <td><h5>pulsar.client.sslProvider</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>The name of the security provider used for SSL connections. The default value is the default security provider of the JVM.</td> + </tr> + <tr> + <td><h5>pulsar.client.statsIntervalSeconds</h5></td> + <td style="word-wrap: break-word;">60</td> + <td>Long</td> + <td>Interval between each stats info.<br /><ul><li>Stats is activated with positive <code class="highlighter-rouge">statsInterval</code></li><li>Set <code class="highlighter-rouge">statsIntervalSeconds</code> to 1 second at least.</li></ul></td> + </tr> + <tr> + <td><h5>pulsar.client.tlsAllowInsecureConnection</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>Whether the Pulsar client accepts untrusted TLS certificate from the broker.</td> + </tr> + <tr> + <td><h5>pulsar.client.tlsCiphers</h5></td> + <td style="word-wrap: break-word;"></td> + <td>List<String></td> + <td>A list of cipher suites. This is a named combination of authentication, encryption, MAC and the key exchange algorithm used to negotiate the security settings for a network connection using the TLS or SSL network protocol. By default all the available cipher suites are supported.</td> + </tr> + <tr> + <td><h5>pulsar.client.tlsHostnameVerificationEnable</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>Whether to enable TLS hostname verification. It allows to validate hostname verification when a client connects to the broker over TLS. It validates incoming x509 certificate and matches provided hostname (CN/SAN) with the expected broker's host name. It follows RFC 2818, 3.1. Server Identity hostname verification.</td> + </tr> + <tr> + <td><h5>pulsar.client.tlsProtocols</h5></td> + <td style="word-wrap: break-word;"></td> + <td>List<String></td> + <td>The SSL protocol used to generate the SSLContext. By default, it is set TLS, which is fine for most cases. Allowed values in recent JVMs are TLS, TLSv1.3, TLSv1.2 and TLSv1.1.</td> + </tr> + <tr> + <td><h5>pulsar.client.tlsTrustCertsFilePath</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>Path to the trusted TLS certificate file.</td> + </tr> + <tr> + <td><h5>pulsar.client.tlsTrustStorePassword</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>The store password for the key store file.</td> + </tr> + <tr> + <td><h5>pulsar.client.tlsTrustStorePath</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>The location of the trust store file.</td> + </tr> + <tr> + <td><h5>pulsar.client.tlsTrustStoreType</h5></td> + <td style="word-wrap: break-word;">"JKS"</td> + <td>String</td> + <td>The file format of the trust store file.</td> + </tr> + <tr> + <td><h5>pulsar.client.useKeyStoreTls</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>If TLS is enabled, whether use the KeyStore type as the TLS configuration parameter. If it is set to <code class="highlighter-rouge">false</code>, it means to use the default pem type configuration.</td> + </tr> + <tr> + <td><h5>pulsar.client.useTcpNoDelay</h5></td> + <td style="word-wrap: break-word;">true</td> + <td>Boolean</td> + <td>Whether to use the TCP no-delay flag on the connection to disable Nagle algorithm.<br />No-delay features ensures that packets are sent out on the network as soon as possible, and it is critical to achieve low latency publishes. On the other hand, sending out a huge number of small packets might limit the overall throughput. Therefore, if latency is not a concern, it is recommended to set this option to <code class="highlighter-rouge">false</code>.<br />By default, it is [...] + </tr> + </tbody> +</table> diff --git a/docs/layouts/shortcodes/generated/pulsar_consumer_configuration.html b/docs/layouts/shortcodes/generated/pulsar_consumer_configuration.html new file mode 100644 index 0000000..e206ff4 --- /dev/null +++ b/docs/layouts/shortcodes/generated/pulsar_consumer_configuration.html @@ -0,0 +1,168 @@ +<table class="configuration table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 20%">Key</th> + <th class="text-left" style="width: 15%">Default</th> + <th class="text-left" style="width: 10%">Type</th> + <th class="text-left" style="width: 55%">Description</th> + </tr> + </thead> + <tbody> + <tr> + <td><h5>pulsar.consumer.ackReceiptEnabled</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>Acknowledgement will return a receipt but this does not mean that the message will not be resent after getting the receipt.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.ackTimeoutMillis</h5></td> + <td style="word-wrap: break-word;">0</td> + <td>Long</td> + <td>The timeout (in ms) for unacknowledged messages, truncated to the nearest millisecond. The timeout needs to be greater than 1 second.<br />By default, the acknowledge timeout is disabled and that means that messages delivered to a consumer will not be re-delivered unless the consumer crashes.<br />When acknowledgement timeout being enabled, if a message is not acknowledged within the specified timeout it will be re-delivered to the consumer (possibly to a different consum [...] + </tr> + <tr> + <td><h5>pulsar.consumer.acknowledgementsGroupTimeMicros</h5></td> + <td style="word-wrap: break-word;">100000</td> + <td>Long</td> + <td>Group a consumer acknowledgment for a specified time (in μs). By default, a consumer uses <code class="highlighter-rouge">100μs</code> grouping time to send out acknowledgments to a broker. If the group time is set to <code class="highlighter-rouge">0</code>, acknowledgments are sent out immediately. A longer ack group time is more efficient at the expense of a slight increase in message re-deliveries after a failure.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.autoAckOldestChunkedMessageOnQueueFull</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>Buffering a large number of outstanding uncompleted chunked messages can bring memory pressure and it can be guarded by providing this <code class="highlighter-rouge">pulsar.consumer.maxPendingChunkedMessage</code> threshold. Once a consumer reaches this threshold, it drops the outstanding unchunked-messages by silently acknowledging if <code class="highlighter-rouge">pulsar.consumer.autoAckOldestChunkedMessageOnQueueFull</code> is true. Otherwise, it marks them for redel [...] + </tr> + <tr> + <td><h5>pulsar.consumer.autoUpdatePartitionsIntervalSeconds</h5></td> + <td style="word-wrap: break-word;">60</td> + <td>Integer</td> + <td>The interval (in seconds) of updating partitions. This only works if autoUpdatePartitions is enabled.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.consumerName</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>The consumer name is informative and it can be used to identify a particular consumer instance from the topic stats.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.cryptoFailureAction</h5></td> + <td style="word-wrap: break-word;">FAIL</td> + <td><p>Enum</p></td> + <td>The consumer should take action when it receives a message that can not be decrypted.<ul><li><code class="highlighter-rouge">FAIL</code>: this is the default option to fail messages until crypto succeeds.</li><li><code class="highlighter-rouge">DISCARD</code>: silently acknowledge but do not deliver messages to an application.</li><li><code class="highlighter-rouge">CONSUME</code>: deliver encrypted messages to applications. It is the application's responsibility to decry [...] + </tr> + <tr> + <td><h5>pulsar.consumer.deadLetterPolicy.deadLetterTopic</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>Name of the dead topic where the failed messages are sent.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.deadLetterPolicy.maxRedeliverCount</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>Integer</td> + <td>The maximum number of times that a message are redelivered before being sent to the dead letter queue.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.deadLetterPolicy.retryLetterTopic</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>Name of the retry topic where the failed messages are sent.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.expireTimeOfIncompleteChunkedMessageMillis</h5></td> + <td style="word-wrap: break-word;">60000</td> + <td>Long</td> + <td>If a producer fails to publish all the chunks of a message, the consumer can expire incomplete chunks if the consumer cannot receive all chunks in expire times (default 1 hour, in ms).</td> + </tr> + <tr> + <td><h5>pulsar.consumer.maxPendingChunkedMessage</h5></td> + <td style="word-wrap: break-word;">10</td> + <td>Integer</td> + <td>The consumer buffers chunk messages into memory until it receives all the chunks of the original message. While consuming chunk-messages, chunks from the same message might not be contiguous in the stream and they might be mixed with other messages' chunks. So, consumer has to maintain multiple buffers to manage chunks coming from different messages. This mainly happens when multiple publishers are publishing messages on the topic concurrently or publishers failed to publ [...] + </tr> + <tr> + <td><h5>pulsar.consumer.maxTotalReceiverQueueSizeAcrossPartitions</h5></td> + <td style="word-wrap: break-word;">50000</td> + <td>Integer</td> + <td>The maximum total receiver queue size across partitions.<br />This setting reduces the receiver queue size for individual partitions if the total receiver queue size exceeds this value.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.negativeAckRedeliveryDelayMicros</h5></td> + <td style="word-wrap: break-word;">60000000</td> + <td>Long</td> + <td>Delay (in μs) to wait before redelivering messages that failed to be processed.<br />When an application uses <code class="highlighter-rouge">Consumer.negativeAcknowledge(Message)</code>, failed messages are redelivered after a fixed timeout.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.poolMessages</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>Enable pooling of messages and the underlying data buffers.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.priorityLevel</h5></td> + <td style="word-wrap: break-word;">0</td> + <td>Integer</td> + <td>Priority level for a consumer to which a broker gives more priorities while dispatching messages in the shared subscription type.<br />The broker follows descending priorities. For example, 0=max-priority, 1, 2,...<br />In shared subscription mode, the broker first dispatches messages to the consumers on the highest priority level if they have permits. Otherwise, the broker considers consumers on the next priority level.<br /><br />Example 1<br />If a subscription has con [...] +C1, 0, 2 +C2, 0, 1 +C3, 0, 1 +C4, 1, 2 +C5, 1, 1 +<br />The order in which a broker dispatches messages to consumers is: C1, C2, C3, C1, C4, C5, C4.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.properties</h5></td> + <td style="word-wrap: break-word;"></td> + <td>Map</td> + <td>A name or value property of this consumer. <code class="highlighter-rouge">properties</code> is application defined metadata attached to a consumer. When getting a topic stats, associate this metadata with the consumer stats for easier identification.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.readCompacted</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>If enabling <code class="highlighter-rouge">readCompacted</code>, a consumer reads messages from a compacted topic rather than reading a full message backlog of a topic.<br />A consumer only sees the latest value for each key in the compacted topic, up until reaching the point in the topic message when compacting backlog. Beyond that point, send messages as normal.<br />Only enabling <code class="highlighter-rouge">readCompacted</code> on subscriptions to persistent topic [...] + </tr> + <tr> + <td><h5>pulsar.consumer.receiverQueueSize</h5></td> + <td style="word-wrap: break-word;">1000</td> + <td>Integer</td> + <td>Size of a consumer's receiver queue.<br />For example, the number of messages accumulated by a consumer before an application calls <code class="highlighter-rouge">Receive</code>.<br />A value higher than the default value increases consumer throughput, though at the expense of more memory utilization.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.replicateSubscriptionState</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>If <code class="highlighter-rouge">replicateSubscriptionState</code> is enabled, a subscription state is replicated to geo-replicated clusters.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.retryEnable</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>If enabled, the consumer will automatically retry messages.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.subscriptionMode</h5></td> + <td style="word-wrap: break-word;">Durable</td> + <td><p>Enum</p></td> + <td>Select the subscription mode to be used when subscribing to the topic.<ul><li><code class="highlighter-rouge">Durable</code>: Make the subscription to be backed by a durable cursor that will retain messages and persist the current position.</li><li><code class="highlighter-rouge">NonDurable</code>: Lightweight subscription mode that doesn't have a durable cursor associated</li></ul><br /><br />Possible values:<ul><li>"Durable"</li><li>"NonDurable"</li></ul></td> + </tr> + <tr> + <td><h5>pulsar.consumer.subscriptionName</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>Specify the subscription name for this consumer. This argument is required when constructing the consumer.</td> + </tr> + <tr> + <td><h5>pulsar.consumer.subscriptionType</h5></td> + <td style="word-wrap: break-word;">Shared</td> + <td><p>Enum</p></td> + <td>Subscription type.<br /><br />Four subscription types are available:<ul><li>Exclusive</li><li>Failover</li><li>Shared</li><li>Key_Shared</li></ul><br /><br />Possible values:<ul><li>"Exclusive"</li><li>"Shared"</li><li>"Failover"</li><li>"Key_Shared"</li></ul></td> + </tr> + <tr> + <td><h5>pulsar.consumer.tickDurationMillis</h5></td> + <td style="word-wrap: break-word;">1000</td> + <td>Long</td> + <td>Granularity (in ms) of the ack-timeout redelivery.<br />A greater (for example, 1 hour) <code class="highlighter-rouge">tickDurationMillis</code> reduces the memory overhead to track messages.</td> + </tr> + </tbody> +</table> diff --git a/docs/layouts/shortcodes/generated/pulsar_producer_configuration.html b/docs/layouts/shortcodes/generated/pulsar_producer_configuration.html new file mode 100644 index 0000000..7984903 --- /dev/null +++ b/docs/layouts/shortcodes/generated/pulsar_producer_configuration.html @@ -0,0 +1,78 @@ +<table class="configuration table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 20%">Key</th> + <th class="text-left" style="width: 15%">Default</th> + <th class="text-left" style="width: 10%">Type</th> + <th class="text-left" style="width: 55%">Description</th> + </tr> + </thead> + <tbody> + <tr> + <td><h5>pulsar.producer.batchingEnabled</h5></td> + <td style="word-wrap: break-word;">true</td> + <td>Boolean</td> + <td>Enable batch send ability, it was enabled by default.</td> + </tr> + <tr> + <td><h5>pulsar.producer.batchingMaxBytes</h5></td> + <td style="word-wrap: break-word;">131072</td> + <td>Integer</td> + <td>The maximum size of messages permitted in a batch. Keep the maximum consistent as previous versions.</td> + </tr> + <tr> + <td><h5>pulsar.producer.batchingMaxMessages</h5></td> + <td style="word-wrap: break-word;">1000</td> + <td>Integer</td> + <td>The maximum number of messages permitted in a batch.</td> + </tr> + <tr> + <td><h5>pulsar.producer.batchingMaxPublishDelayMicros</h5></td> + <td style="word-wrap: break-word;">1000</td> + <td>Long</td> + <td>Batching time period of sending messages.</td> + </tr> + <tr> + <td><h5>pulsar.producer.batchingPartitionSwitchFrequencyByPublishDelay</h5></td> + <td style="word-wrap: break-word;">10</td> + <td>Integer</td> + <td>The maximum wait time for switching topic partitions.</td> + </tr> + <tr> + <td><h5>pulsar.producer.chunkingEnabled</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td></td> + </tr> + <tr> + <td><h5>pulsar.producer.compressionType</h5></td> + <td style="word-wrap: break-word;">NONE</td> + <td><p>Enum</p></td> + <td>Message data compression type used by a producer.Available options:<ul><li><a href="LZ4">https://github.com/lz4/lz4</a></li><li><a href="ZLIB">https://zlib.net/</a></li><li><a href="ZSTD">https://facebook.github.io/zstd/</a></li><li><a href="SNAPPY">https://google.github.io/snappy/</a></li></ul><br /><br />Possible values:<ul><li>"NONE"</li><li>"LZ4"</li><li>"ZLIB"</li><li>"ZSTD"</li><li>"SNAPPY"</li></ul></td> + </tr> + <tr> + <td><h5>pulsar.producer.initialSequenceId</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>Long</td> + <td>The sequence id for avoiding the duplication, it's used when Pulsar doesn't have transaction.</td> + </tr> + <tr> + <td><h5>pulsar.producer.producerName</h5></td> + <td style="word-wrap: break-word;">(none)</td> + <td>String</td> + <td>A producer name which would be displayed in the Pulsar's dashboard. If no producer name was provided, we would use a Pulsar generated name instead.</td> + </tr> + <tr> + <td><h5>pulsar.producer.properties</h5></td> + <td style="word-wrap: break-word;"></td> + <td>Map</td> + <td>A name or value property of this consumer. <code class="highlighter-rouge">properties</code> is application defined metadata attached to a consumer. When getting a topic stats, associate this metadata with the consumer stats for easier identification.</td> + </tr> + <tr> + <td><h5>pulsar.producer.sendTimeoutMs</h5></td> + <td style="word-wrap: break-word;">30000</td> + <td>Long</td> + <td>Message send timeout in ms.If a message is not acknowledged by a server before the <code class="highlighter-rouge">sendTimeout</code> expires, an error occurs.</td> + </tr> + </tbody> +</table> diff --git a/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html b/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html new file mode 100644 index 0000000..c1ff77e --- /dev/null +++ b/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html @@ -0,0 +1,54 @@ +<table class="configuration table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 20%">Key</th> + <th class="text-left" style="width: 15%">Default</th> + <th class="text-left" style="width: 10%">Type</th> + <th class="text-left" style="width: 55%">Description</th> + </tr> + </thead> + <tbody> + <tr> + <td><h5>pulsar.sink.deliveryGuarantee</h5></td> + <td style="word-wrap: break-word;">none</td> + <td><p>Enum</p></td> + <td>Optional delivery guarantee when committing.<br /><br />Possible values:<ul><li>"exactly-once": Records are only delivered exactly-once also under failover scenarios. To build a complete exactly-once pipeline is required that the source and sink support exactly-once and are properly configured.</li><li>"at-least-once": Records are ensured to be delivered but it may happen that the same record is delivered multiple times. Usually, this guarantee is faster than the exactly- [...] + </tr> + <tr> + <td><h5>pulsar.sink.enableSchemaEvolution</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>If you enable this option and use PulsarSerializationSchema.pulsarSchema(), we would consume and deserialize the message by using Pulsar's <code class="highlighter-rouge">Schema</code>.</td> + </tr> + <tr> + <td><h5>pulsar.sink.maxPendingMessages</h5></td> + <td style="word-wrap: break-word;">1000</td> + <td>Integer</td> + <td>The maximum number of pending messages in one sink parallelism.</td> + </tr> + <tr> + <td><h5>pulsar.sink.maxRecommitTimes</h5></td> + <td style="word-wrap: break-word;">5</td> + <td>Integer</td> + <td>The allowed transaction recommit times if we meet some retryable exception. This is used in Pulsar Transaction.</td> + </tr> + <tr> + <td><h5>pulsar.sink.messageKeyHash</h5></td> + <td style="word-wrap: break-word;">murmur-3-32-hash</td> + <td><p>Enum</p></td> + <td>The hash policy for routing message by calculating the hash code of message key.<br /><br />Possible values:<ul><li>"java-hash": This hash would use <code class="highlighter-rouge">String.hashCode()</code> to calculate the message key string's hash code.</li><li>"murmur-3-32-hash": This hash would calculate message key's hash code by using <a href="https://en.wikipedia.org/wiki/MurmurHash";>Murmur3</a> algorithm.</li></ul></td> + </tr> + <tr> + <td><h5>pulsar.sink.topicMetadataRefreshInterval</h5></td> + <td style="word-wrap: break-word;">1800000</td> + <td>Long</td> + <td>Auto update the topic metadata in a fixed interval (in ms). The default value is 30 minutes.</td> + </tr> + <tr> + <td><h5>pulsar.sink.transactionTimeoutMillis</h5></td> + <td style="word-wrap: break-word;">10800000</td> + <td>Long</td> + <td>This option is used when the user require the <code class="highlighter-rouge">DeliveryGuarantee.EXACTLY_ONCE</code> semantic.We would use transaction for making sure the message could be write only once.</td> + </tr> + </tbody> +</table> diff --git a/docs/layouts/shortcodes/generated/pulsar_source_configuration.html b/docs/layouts/shortcodes/generated/pulsar_source_configuration.html new file mode 100644 index 0000000..65053e4 --- /dev/null +++ b/docs/layouts/shortcodes/generated/pulsar_source_configuration.html @@ -0,0 +1,60 @@ +<table class="configuration table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 20%">Key</th> + <th class="text-left" style="width: 15%">Default</th> + <th class="text-left" style="width: 10%">Type</th> + <th class="text-left" style="width: 55%">Description</th> + </tr> + </thead> + <tbody> + <tr> + <td><h5>pulsar.source.allowKeySharedOutOfOrderDelivery</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>If enabled, it will relax the ordering requirement, allowing the broker to send out-of-order messages in case of failures. This will make it faster for new consumers to join without being stalled by an existing slow consumer.<br />In this case, a single consumer will still receive all the keys, but they may be coming in different orders.</td> + </tr> + <tr> + <td><h5>pulsar.source.autoCommitCursorInterval</h5></td> + <td style="word-wrap: break-word;">5000</td> + <td>Long</td> + <td>This option is used only when the user disables the checkpoint and uses Exclusive or Failover subscription. We would automatically commit the cursor using the given period (in ms).</td> + </tr> + <tr> + <td><h5>pulsar.source.enableAutoAcknowledgeMessage</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>Flink commits the consuming position with pulsar transactions on checkpoint. However, if you have disabled the Flink checkpoint or disabled transaction for your Pulsar cluster, ensure that you have set this option to <code class="highlighter-rouge">true</code>.<br />The source would use pulsar client's internal mechanism and commit cursor in two ways.<ul><li>For <code class="highlighter-rouge">Key_Shared</code> and <code class="highlighter-rouge">Shared</code> subscriptio [...] + </tr> + <tr> + <td><h5>pulsar.source.maxFetchRecords</h5></td> + <td style="word-wrap: break-word;">100</td> + <td>Integer</td> + <td>The maximum number of records to fetch to wait when polling. A longer time increases throughput but also latency. A fetch batch might be finished earlier because of <code class="highlighter-rouge">pulsar.source.maxFetchTime</code>.</td> + </tr> + <tr> + <td><h5>pulsar.source.maxFetchTime</h5></td> + <td style="word-wrap: break-word;">10000</td> + <td>Long</td> + <td>The maximum time (in ms) to wait when fetching records. A longer time increases throughput but also latency. A fetch batch might be finished earlier because of <code class="highlighter-rouge">pulsar.source.maxFetchRecords</code>.</td> + </tr> + <tr> + <td><h5>pulsar.source.partitionDiscoveryIntervalMs</h5></td> + <td style="word-wrap: break-word;">30000</td> + <td>Long</td> + <td>The interval (in ms) for the Pulsar source to discover the new partitions. A non-positive value disables the partition discovery.</td> + </tr> + <tr> + <td><h5>pulsar.source.transactionTimeoutMillis</h5></td> + <td style="word-wrap: break-word;">10800000</td> + <td>Long</td> + <td>This option is used in <code class="highlighter-rouge">Shared</code> or <code class="highlighter-rouge">Key_Shared</code> subscription. You should configure this option when you do not enable the <code class="highlighter-rouge">pulsar.source.enableAutoAcknowledgeMessage</code> option.<br />The value (in ms) should be greater than the checkpoint interval.</td> + </tr> + <tr> + <td><h5>pulsar.source.verifyInitialOffsets</h5></td> + <td style="word-wrap: break-word;">WARN_ON_MISMATCH</td> + <td><p>Enum</p></td> + <td>Upon (re)starting the source, check whether the expected message can be read. If failure is enabled, the application fails. Otherwise, it logs a warning. A possible solution is to adjust the retention settings in Pulsar or ignoring the check result.<br /><br />Possible values:<ul><li>"FAIL_ON_MISMATCH": Fail the consuming from Pulsar when we don't find the related cursor.</li><li>"WARN_ON_MISMATCH": Print a warn message and start consuming from the valid offset.</li></ul></td> + </tr> + </tbody> +</table>
