[GitHub] [pulsar] sijie commented on issue #3804: ManagedLedger should streamline the read requests
sijie commented on issue #3804: ManagedLedger should streamline the read requests URL: https://github.com/apache/pulsar/issues/3804#issuecomment-531535834 @vicaya thank you for your feedback. @MarvinCai are you willing to give it a try? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam commented on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription
massakam commented on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription URL: https://github.com/apache/pulsar/pull/5185#issuecomment-531533009 rerun java8 tests rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam commented on issue #5193: [pulsar-broker] Make some member variables of Dispatcher volatile
massakam commented on issue #5193: [pulsar-broker] Make some member variables of Dispatcher volatile URL: https://github.com/apache/pulsar/pull/5193#issuecomment-531532991 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam commented on issue #5191: [Issue #5176][pulsar-broker] Fix bug that fails to search namespace bundle due to NPE
massakam commented on issue #5191: [Issue #5176][pulsar-broker] Fix bug that fails to search namespace bundle due to NPE URL: https://github.com/apache/pulsar/pull/5191#issuecomment-531532977 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] vicaya commented on issue #3804: ManagedLedger should streamline the read requests
vicaya commented on issue #3804: ManagedLedger should streamline the read requests URL: https://github.com/apache/pulsar/issues/3804#issuecomment-531530419 This is a also blocking issue for practical use of tiered storage as historical retention, since replay from tiered storage (at least s3) is too slow. It'd be great if number of read-ahead threads and/or outstanding requests is tunable, as many blob/object store scale throughput proportional to number of connections. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on a change in pull request #5181: [Doc] Update *Develop Connectors Guide*
tuteng commented on a change in pull request #5181: [Doc] Update *Develop
Connectors Guide*
URL: https://github.com/apache/pulsar/pull/5181#discussion_r324441631
##
File path: site2/docs/io-develop.md
##
@@ -1,156 +1,188 @@
---
id: io-develop
-title: Develop Connectors
-sidebar_label: Developing Connectors
+title: How to develop Pulsar connectors
+sidebar_label: Develop
---
-This guide describes how developers can write new connectors for Pulsar IO to
move data
-between Pulsar and other systems. It describes how to create a Pulsar IO
connector.
+This guide describes how to develop Pulsar connectors to move data
+between Pulsar and other systems.
-Pulsar IO connectors are specialized [Pulsar
Functions](functions-overview.md). So writing
-a Pulsar IO connector is as simple as writing a Pulsar function. Pulsar IO
connectors come
-in two flavors: {@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java},
-which import data from another system, and {@inject:
github:`Sink`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java},
-which export data to another system. For example, [KinesisSink](io-kinesis.md)
would export
-the messages of a Pulsar topic to a Kinesis stream, and
[RabbitmqSource](io-rabbitmq.md) would import
-the messages of a RabbitMQ queue to a Pulsar topic.
+Pulsar connectors are special [Pulsar Functions](functions-overview.md), so
creating
+a Pulsar connector is similar to creating a Pulsar function.
-### Developing
+Pulsar connectors come in two types:
- Develop a source connector
+| Type | Description | Example
+|---|---|---
+{@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import
data from another system to Pulsar.|[RabbitMQ source
connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar
topic.
+{@inject:
github:`Sink`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export
data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md)
exports the messages of a Pulsar topic to a Kinesis stream.
-What you need to develop a source connector is to implement {@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}
-interface.
+## Develop
-First, you need to implement the {@inject:
github:`open`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L33}
method. This method will be called once when the source connector
-is initialized. In this method, you can retrieve all the connector specific
settings through
-the passed `config` parameter, and initialize all the necessary resourcess.
For example, a Kafka
-connector can create the Kafka client in this `open` method.
+You can develop Pulsar source connectors and sink connectors.
-Beside the passed-in `config` object, the Pulsar runtime also provides a
`SourceContext` for the
-connector to access runtime resources for tasks like collecting metrics. The
implementation can
-save the `SourceContext` for futher usage.
+### Source
-```java
+Developing a source connector is to implement the {@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}
+interface, which means you need to implement the {@inject:
github:`open`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L33}
method and the {@inject:
github:`record`:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java#L28}
method.
+
+1. Implement the {@inject:
github:`open`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L33}
method.
+
+```java
/**
- * Open connector with configuration
- *
- * @param config initialization config
- * @param sourceContext
- * @throws Exception IO type exceptions when opening a connector
- */
+* Open connector with configuration
+*
+* @param config initialization config
+* @param sourceContext
+* @throws Exception IO type exceptions when opening a connector
+*/
void open(final Map config, SourceContext sourceContext)
throws Exception;
-```
+```
-The main task for a Source implementor is to implement {@inject:
github:`read`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L41}
-method.
+This method is called when the source connector is initialized.
-```java
-/**
- * Reads the next message from source.
- * If source does not have any new messages, this call should block.
- * @return next message from source. The return result should never be
null
- * @throws Exception
- */
-Record read() throws Exception;
-```
+In this method, you can retrieve all connector specific settings through
the passed-in `config` parameter and initialize all necessary resources.
+
+For example, a Kafka connector can create a Kafka client in this `open`
method.
-The
[GitHub] [pulsar] tuteng commented on a change in pull request #5181: [Doc] Update *Develop Connectors Guide*
tuteng commented on a change in pull request #5181: [Doc] Update *Develop
Connectors Guide*
URL: https://github.com/apache/pulsar/pull/5181#discussion_r324441476
##
File path: site2/docs/io-develop.md
##
@@ -1,156 +1,188 @@
---
id: io-develop
-title: Develop Connectors
-sidebar_label: Developing Connectors
+title: How to develop Pulsar connectors
+sidebar_label: Develop
---
-This guide describes how developers can write new connectors for Pulsar IO to
move data
-between Pulsar and other systems. It describes how to create a Pulsar IO
connector.
+This guide describes how to develop Pulsar connectors to move data
+between Pulsar and other systems.
-Pulsar IO connectors are specialized [Pulsar
Functions](functions-overview.md). So writing
-a Pulsar IO connector is as simple as writing a Pulsar function. Pulsar IO
connectors come
-in two flavors: {@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java},
-which import data from another system, and {@inject:
github:`Sink`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java},
-which export data to another system. For example, [KinesisSink](io-kinesis.md)
would export
-the messages of a Pulsar topic to a Kinesis stream, and
[RabbitmqSource](io-rabbitmq.md) would import
-the messages of a RabbitMQ queue to a Pulsar topic.
+Pulsar connectors are special [Pulsar Functions](functions-overview.md), so
creating
+a Pulsar connector is similar to creating a Pulsar function.
-### Developing
+Pulsar connectors come in two types:
- Develop a source connector
+| Type | Description | Example
+|---|---|---
+{@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import
data from another system to Pulsar.|[RabbitMQ source
connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar
topic.
+{@inject:
github:`Sink`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export
data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md)
exports the messages of a Pulsar topic to a Kinesis stream.
-What you need to develop a source connector is to implement {@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}
-interface.
+## Develop
-First, you need to implement the {@inject:
github:`open`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L33}
method. This method will be called once when the source connector
-is initialized. In this method, you can retrieve all the connector specific
settings through
-the passed `config` parameter, and initialize all the necessary resourcess.
For example, a Kafka
-connector can create the Kafka client in this `open` method.
+You can develop Pulsar source connectors and sink connectors.
-Beside the passed-in `config` object, the Pulsar runtime also provides a
`SourceContext` for the
-connector to access runtime resources for tasks like collecting metrics. The
implementation can
-save the `SourceContext` for futher usage.
+### Source
-```java
+Developing a source connector is to implement the {@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}
+interface, which means you need to implement the {@inject:
github:`open`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L33}
method and the {@inject:
github:`record`:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java#L28}
method.
+
+1. Implement the {@inject:
github:`open`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L33}
method.
+
+```java
/**
- * Open connector with configuration
- *
- * @param config initialization config
- * @param sourceContext
- * @throws Exception IO type exceptions when opening a connector
- */
+* Open connector with configuration
+*
+* @param config initialization config
+* @param sourceContext
+* @throws Exception IO type exceptions when opening a connector
+*/
void open(final Map config, SourceContext sourceContext)
throws Exception;
-```
+```
-The main task for a Source implementor is to implement {@inject:
github:`read`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L41}
-method.
+This method is called when the source connector is initialized.
-```java
-/**
- * Reads the next message from source.
- * If source does not have any new messages, this call should block.
- * @return next message from source. The return result should never be
null
- * @throws Exception
- */
-Record read() throws Exception;
-```
+In this method, you can retrieve all connector specific settings through
the passed-in `config` parameter and initialize all necessary resources.
+
+For example, a Kafka connector can create a Kafka client in this `open`
method.
-The
[GitHub] [pulsar] tuteng commented on a change in pull request #5181: [Doc] Update *Develop Connectors Guide*
tuteng commented on a change in pull request #5181: [Doc] Update *Develop
Connectors Guide*
URL: https://github.com/apache/pulsar/pull/5181#discussion_r324441602
##
File path: site2/docs/io-develop.md
##
@@ -1,156 +1,188 @@
---
id: io-develop
-title: Develop Connectors
-sidebar_label: Developing Connectors
+title: How to develop Pulsar connectors
+sidebar_label: Develop
---
-This guide describes how developers can write new connectors for Pulsar IO to
move data
-between Pulsar and other systems. It describes how to create a Pulsar IO
connector.
+This guide describes how to develop Pulsar connectors to move data
+between Pulsar and other systems.
-Pulsar IO connectors are specialized [Pulsar
Functions](functions-overview.md). So writing
-a Pulsar IO connector is as simple as writing a Pulsar function. Pulsar IO
connectors come
-in two flavors: {@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java},
-which import data from another system, and {@inject:
github:`Sink`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java},
-which export data to another system. For example, [KinesisSink](io-kinesis.md)
would export
-the messages of a Pulsar topic to a Kinesis stream, and
[RabbitmqSource](io-rabbitmq.md) would import
-the messages of a RabbitMQ queue to a Pulsar topic.
+Pulsar connectors are special [Pulsar Functions](functions-overview.md), so
creating
+a Pulsar connector is similar to creating a Pulsar function.
-### Developing
+Pulsar connectors come in two types:
- Develop a source connector
+| Type | Description | Example
+|---|---|---
+{@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import
data from another system to Pulsar.|[RabbitMQ source
connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar
topic.
+{@inject:
github:`Sink`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export
data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md)
exports the messages of a Pulsar topic to a Kinesis stream.
-What you need to develop a source connector is to implement {@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}
-interface.
+## Develop
-First, you need to implement the {@inject:
github:`open`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L33}
method. This method will be called once when the source connector
-is initialized. In this method, you can retrieve all the connector specific
settings through
-the passed `config` parameter, and initialize all the necessary resourcess.
For example, a Kafka
-connector can create the Kafka client in this `open` method.
+You can develop Pulsar source connectors and sink connectors.
-Beside the passed-in `config` object, the Pulsar runtime also provides a
`SourceContext` for the
-connector to access runtime resources for tasks like collecting metrics. The
implementation can
-save the `SourceContext` for futher usage.
+### Source
-```java
+Developing a source connector is to implement the {@inject:
github:`Source`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}
+interface, which means you need to implement the {@inject:
github:`open`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L33}
method and the {@inject:
github:`record`:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java#L28}
method.
+
+1. Implement the {@inject:
github:`open`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L33}
method.
+
+```java
/**
- * Open connector with configuration
- *
- * @param config initialization config
- * @param sourceContext
- * @throws Exception IO type exceptions when opening a connector
- */
+* Open connector with configuration
+*
+* @param config initialization config
+* @param sourceContext
+* @throws Exception IO type exceptions when opening a connector
+*/
void open(final Map config, SourceContext sourceContext)
throws Exception;
-```
+```
-The main task for a Source implementor is to implement {@inject:
github:`read`:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java#L41}
-method.
+This method is called when the source connector is initialized.
-```java
-/**
- * Reads the next message from source.
- * If source does not have any new messages, this call should block.
- * @return next message from source. The return result should never be
null
- * @throws Exception
- */
-Record read() throws Exception;
-```
+In this method, you can retrieve all connector specific settings through
the passed-in `config` parameter and initialize all necessary resources.
+
+For example, a Kafka connector can create a Kafka client in this `open`
method.
-The
[GitHub] [pulsar] tuteng commented on issue #5184: [Doc] Move Connector Admin CLI to Connector Guide
tuteng commented on issue #5184: [Doc] Move Connector Admin CLI to Connector Guide URL: https://github.com/apache/pulsar/pull/5184#issuecomment-531521857 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #5184: [Doc] Move Connector Admin CLI to Connector Guide
tuteng commented on issue #5184: [Doc] Move Connector Admin CLI to Connector Guide URL: https://github.com/apache/pulsar/pull/5184#issuecomment-531521865 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on a change in pull request #5183: [Doc] Update *Overview of Connector*
tuteng commented on a change in pull request #5183: [Doc] Update *Overview of Connector* URL: https://github.com/apache/pulsar/pull/5183#discussion_r324441262 ## File path: site2/docs/io-overview.md ## @@ -1,40 +1,45 @@ --- id: io-overview -title: Pulsar IO Overview +title: Pulsar connector overview sidebar_label: Overview --- -Messaging systems are most powerful when you can easily use them in conjunction with external systems like databases and other messaging systems. **Pulsar IO** is a feature of Pulsar that enables you to easily create, deploy, and manage Pulsar **connectors** that interact with external systems, such as [Apache Cassandra](https://cassandra.apache.org), [Aerospike](https://www.aerospike.com), and many others. +Messaging systems are most powerful when you can easily use them with external systems like databases and other messaging systems. -> Pulsar IO and Pulsar Functions -> Under the hood, Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md) purpose-built to interface with external systems. The [administrative interface](io-quickstart.md) for Pulsar IO is, in fact, quite similar to that of Pulsar Functions. +**Pulsar IO connectors** enable you to easily create, deploy, and manage connectors that interact with external systems, such as [Apache Cassandra](https://cassandra.apache.org), [Aerospike](https://www.aerospike.com), and many others. -## Sources and sinks -Pulsar IO connectors come in two types: +## Concept -* **Sources** feed data *into* Pulsar from other systems. Common sources include other messaging systems and "firehose"-style data pipeline APIs. -* **Sinks** are fed data *from* Pulsar. Common sinks include other messaging systems and SQL and NoSQL databases. +Pulsar IO connectors come in two types: **source** and **sink**. -This diagram illustrates the relationship between sources, sinks, and Pulsar: +This diagram illustrates the relationship between source, Pulsar, and sink:  -## Working with connectors +### Source -Pulsar IO connectors can be managed via the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands. +> Sources **feed data from external systems into Pulsar**. -> For a guide to managing connectors in your Pulsar installation, see the [Getting started with Pulsar IO](io-quickstart.md) +Common sources include other messaging systems and firehose-style data pipeline APIs. -The following connectors are currently available for Pulsar: +For the complete list of Pulsar built-in source connectors, see [source connector](io-connectors.md#source-connector). + +### Sink + +> Sinks **feed data from Pulsar into external systems**. + +Common sinks include other messaging systems and SQL and NoSQL databases. + +For the complete list of Pulsar built-in sink connectors, see [sink connector](io-connectors.md#sink-connector). + +## Work with connector + +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. + +## Relationship between connector and Functions + +Pulsar connectors are **special** [Pulsar Functions](functions-overview.md) built to interact with external systems. + +Sources, sinks, and Functions are components of instances, and they all run on Functions workers. When managing sources, sinks or Functions via **Pulsar admin CLI** (that is, [Connector CLI](reference-connector-admin.md) and [Functions CLI](functions-cli.md)), an instance is started on a worker. Review comment: `sources, sinks or Functions` does the first letter have to be uppercase or lowercase? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on a change in pull request #5183: [Doc] Update *Overview of Connector*
tuteng commented on a change in pull request #5183: [Doc] Update *Overview of Connector* URL: https://github.com/apache/pulsar/pull/5183#discussion_r324441232 ## File path: site2/docs/io-overview.md ## @@ -1,40 +1,45 @@ --- id: io-overview -title: Pulsar IO Overview +title: Pulsar connector overview sidebar_label: Overview --- -Messaging systems are most powerful when you can easily use them in conjunction with external systems like databases and other messaging systems. **Pulsar IO** is a feature of Pulsar that enables you to easily create, deploy, and manage Pulsar **connectors** that interact with external systems, such as [Apache Cassandra](https://cassandra.apache.org), [Aerospike](https://www.aerospike.com), and many others. +Messaging systems are most powerful when you can easily use them with external systems like databases and other messaging systems. -> Pulsar IO and Pulsar Functions -> Under the hood, Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md) purpose-built to interface with external systems. The [administrative interface](io-quickstart.md) for Pulsar IO is, in fact, quite similar to that of Pulsar Functions. +**Pulsar IO connectors** enable you to easily create, deploy, and manage connectors that interact with external systems, such as [Apache Cassandra](https://cassandra.apache.org), [Aerospike](https://www.aerospike.com), and many others. -## Sources and sinks -Pulsar IO connectors come in two types: +## Concept -* **Sources** feed data *into* Pulsar from other systems. Common sources include other messaging systems and "firehose"-style data pipeline APIs. -* **Sinks** are fed data *from* Pulsar. Common sinks include other messaging systems and SQL and NoSQL databases. +Pulsar IO connectors come in two types: **source** and **sink**. -This diagram illustrates the relationship between sources, sinks, and Pulsar: +This diagram illustrates the relationship between source, Pulsar, and sink:  -## Working with connectors +### Source -Pulsar IO connectors can be managed via the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands. +> Sources **feed data from external systems into Pulsar**. -> For a guide to managing connectors in your Pulsar installation, see the [Getting started with Pulsar IO](io-quickstart.md) +Common sources include other messaging systems and firehose-style data pipeline APIs. -The following connectors are currently available for Pulsar: +For the complete list of Pulsar built-in source connectors, see [source connector](io-connectors.md#source-connector). + +### Sink + +> Sinks **feed data from Pulsar into external systems**. + +Common sinks include other messaging systems and SQL and NoSQL databases. + +For the complete list of Pulsar built-in sink connectors, see [sink connector](io-connectors.md#sink-connector). + +## Work with connector + +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. + +## Relationship between connector and Functions + +Pulsar connectors are **special** [Pulsar Functions](functions-overview.md) built to interact with external systems. + +Sources, sinks, and Functions are components of instances, and they all run on Functions workers. When managing sources, sinks or Functions via **Pulsar admin CLI** (that is, [Connector CLI](reference-connector-admin.md) and [Functions CLI](functions-cli.md)), an instance is started on a worker. Review comment: There is no necessary relationship between `connectors` and `functions`. The relationship between `connectors` and `function worker` can be referred to this figure http://pulsar.apache.org/docs/en/functions-worker/#run-functions-worker-separately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] vicaya opened a new pull request #5194: skip-all is clear-backlog after topics unification
vicaya opened a new pull request #5194: skip-all is clear-backlog after topics unification URL: https://github.com/apache/pulsar/pull/5194 Resolves #5060 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam commented on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription
massakam commented on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription URL: https://github.com/apache/pulsar/pull/5185#issuecomment-531504635 rerun java8 tests rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam commented on issue #5191: [Issue #5176][pulsar-broker] Fix bug that fails to search namespace bundle due to NPE
massakam commented on issue #5191: [Issue #5176][pulsar-broker] Fix bug that fails to search namespace bundle due to NPE URL: https://github.com/apache/pulsar/pull/5191#issuecomment-531504563 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam removed a comment on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription
massakam removed a comment on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription URL: https://github.com/apache/pulsar/pull/5185#issuecomment-531486654 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam commented on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription
massakam commented on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription URL: https://github.com/apache/pulsar/pull/5185#issuecomment-531493819 rerun java8 tests rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam opened a new pull request #5193: [pulsar-broker] Make some member variables of Dispatcher volatile
massakam opened a new pull request #5193: [pulsar-broker] Make some member variables of Dispatcher volatile URL: https://github.com/apache/pulsar/pull/5193 Some of the variables of the `Dispatcher` classes seem to should be volatile, so I fixed them. For example, the variable `havePendingRead` is corrently volatile in `PersistentDispatcherSingleActiveConsumer` but not in` PersistentDispatcherMultipleConsumers`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] cdbartholomew commented on issue #5079: Use "info" as the default root logger level
cdbartholomew commented on issue #5079: Use "info" as the default root logger
level
URL: https://github.com/apache/pulsar/pull/5079#issuecomment-531491624
I believe that this new default was also triggering a bug
[3113](https://github.com/eclipse/jetty.project/issues/3133) in jetty: "Logging
of `key.readyOps()` can throw unchecked `CancelledKeyException`"
This bug is only exposed if debug logging is enabled:
`if (LOG.isDebugEnabled())
LOG.debug("selected {} {} {}
",key.readyOps(),key,attachment);
`
I was seeing this in the broker after sending lots of REST requests. After
hitting this bug, the broker no longer responds to REST requests, so it's
pretty ugly.
Once I changed root level logging to info, I no longer see the jetty bug.
I think this is an important change to get in the next milestone so people
don't trip over this. It's also something to note can happen with debug logging
enabled the broker even when this default is changed, so it might be worth
considering upgrading the jetty version. The jetty bug is fixed in 9.4.15
(current Pulsar version is 9.4.12).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] massakam commented on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription
massakam commented on issue #5185: [pulsar-broker] Fix bug that message delivery stops after resetting cursor for failover subscription URL: https://github.com/apache/pulsar/pull/5185#issuecomment-531486654 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #5191: [Issue #5176][pulsar-broker] Fix bug that fails to search namespace bundle due to NPE
jiazhai commented on issue #5191: [Issue #5176][pulsar-broker] Fix bug that fails to search namespace bundle due to NPE URL: https://github.com/apache/pulsar/pull/5191#issuecomment-531484239 run java8 tests run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai merged pull request #5190: [Doc]Fix tenant typo for debezim mysql source
jiazhai merged pull request #5190: [Doc]Fix tenant typo for debezim mysql source URL: https://github.com/apache/pulsar/pull/5190 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai closed issue #5189: 2.4.1 - Pulsar standalone + debezium mysql do not work
jiazhai closed issue #5189: 2.4.1 - Pulsar standalone + debezium mysql do not work URL: https://github.com/apache/pulsar/issues/5189 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: Fix tenant typo from pubilc to public (#5190)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new bbdf14b Fix tenant typo from pubilc to public (#5190) bbdf14b is described below commit bbdf14bd66305758ba5731cef55f4d2547f009cc Author: tuteng AuthorDate: Sat Sep 14 22:30:00 2019 +0800 Fix tenant typo from pubilc to public (#5190) Master Issue: #5189 Motivation Currently, the tenant name for the new version of the document is incorrectly written, resulting in that the tenant cannot be found. Modifications Fix tenant name from pubilc to public --- site2/docs/io-cdc-debezium.md | 2 +- site2/website/versioned_docs/version-2.4.1/io-cdc-debezium.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/site2/docs/io-cdc-debezium.md b/site2/docs/io-cdc-debezium.md index 573548b..5aa3884 100644 --- a/site2/docs/io-cdc-debezium.md +++ b/site2/docs/io-cdc-debezium.md @@ -56,7 +56,7 @@ Here is a JSON configuration example: Optionally, you can create a `debezium-mysql-source-config.yaml` file, and copy the [contents] (https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/resources/debezium-mysql-source-config.yaml) below to the `debezium-mysql-source-config.yaml` file. ```$yaml -tenant: "pubilc" +tenant: "public" namespace: "default" name: "debezium-mysql-source" topicName: "debezium-mysql-topic" diff --git a/site2/website/versioned_docs/version-2.4.1/io-cdc-debezium.md b/site2/website/versioned_docs/version-2.4.1/io-cdc-debezium.md index f2978bd..97e29c4 100644 --- a/site2/website/versioned_docs/version-2.4.1/io-cdc-debezium.md +++ b/site2/website/versioned_docs/version-2.4.1/io-cdc-debezium.md @@ -57,7 +57,7 @@ Here is a JSON configuration example: Optionally, you can create a `debezium-mysql-source-config.yaml` file, and copy the [contents] (https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/resources/debezium-mysql-source-config.yaml) below to the `debezium-mysql-source-config.yaml` file. ```$yaml -tenant: "pubilc" +tenant: "public" namespace: "default" name: "debezium-mysql-source" topicName: "debezium-mysql-topic"
[GitHub] [pulsar] yittg commented on a change in pull request #5165: [PIP-43] Support producer to send msg with different schema
yittg commented on a change in pull request #5165: [PIP-43] Support producer to send msg with different schema URL: https://github.com/apache/pulsar/pull/5165#discussion_r324424636 ## File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Producer.java ## @@ -107,10 +107,19 @@ */ TypedMessageBuilder newMessage(); +/** + * Create a new message builder with schema specified explicitly, + * + * @return a typed message builder that can be used to construct the message to be sent through this producer + * @since 2.5.0 + * @see #newMessage() + */ +TypedMessageBuilder newMessage(Schema schema); Review comment: I have hidden this interface and left a TODO. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] yittg commented on a change in pull request #5165: [PIP-43] Support producer to send msg with different schema
yittg commented on a change in pull request #5165: [PIP-43] Support producer to send msg with different schema URL: https://github.com/apache/pulsar/pull/5165#discussion_r324424596 ## File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java ## @@ -434,4 +434,18 @@ * @return the producer builder instance */ ProducerBuilder autoUpdatePartitions(boolean autoUpdate); + +/** + * Option to control whether enable multiple schema mode, + * which would be enabled automatically if this feature is touched by default. + * If enabled, producer can send a message with different schema from that specified just when it is created, + * otherwise a invalid message exception would be threw if the producer want to send a message + * with different schema. + * + * @param switchOption + *indicates to enable, disable, or automatically enable multiple schema mode + * @return the producer builder instance + * @since 2.5.0 + */ +ProducerBuilder withMultiSchema(SwitchOption switchOption); Review comment: fixed, PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #5192: [doc]Fix javadoc api
tuteng commented on issue #5192: [doc]Fix javadoc api URL: https://github.com/apache/pulsar/pull/5192#issuecomment-531473781 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] vzhikserg commented on issue #5115: Add checkstyle validation and fix style violations in the pulsar client api module
vzhikserg commented on issue #5115: Add checkstyle validation and fix style violations in the pulsar client api module URL: https://github.com/apache/pulsar/pull/5115#issuecomment-531470106 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng opened a new pull request #5192: [doc]Fix javadoc api
tuteng opened a new pull request #5192: [doc]Fix javadoc api URL: https://github.com/apache/pulsar/pull/5192 Fixes https://github.com/apache/pulsar/issues/4979 Master Issue: https://github.com/apache/pulsar/issues/4979 ### Motivation This tool script https://github.com/apache/pulsar/blob/master/site2/tools/javadoc-gen.sh throws a lot of exceptions when it is running, resulting in many interface documents not being generated correctly. I will try to replace it with mvn javadoc:javadoc ### Modifications * Replace `javadoc` use `mvn javadoc:javadoc` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
