cbornet opened a new issue, #17719: URL: https://github.com/apache/pulsar/issues/17719
### Motivation Currently, when you want to consume from Pulsar topics in applications written in languages that don't have a Pulsar driver supported, you need to run some type of proxy like the WebSocket Proxy or Pulsar Beam. In production this needs additional effort to deploy, scale, load balance, monitor, and so on... Pulsar IO is a framework that deals with all these operational subjects and can be leveraged to provide a way to push messages to external systems using HTTP, a protocol supported by every existing language and OS. ### Goal This proposal defines an HTTP Sink that sends the messages to a configured URL. It takes inspiration from [Pulsar Beam](https://github.com/kafkaesque-io/pulsar-beam) and the [Confluent HTTP Sink connector](https://docs.confluent.io/kafka-connectors/http/current/overview.html). ### API Changes _No response_ ### Implementation A `pulsar-io-http` module will be added to `pulsar-io`. On building the project `pulsar-io-http-{version}.nar` will be built and added to the `pulsar-all` distribution. The name of the Sink will be `http`. The HTTP Sink pushes records to any HTTP server with the record value in the body of a POST method. The body of the HTTP request is the JSON representation of the record value. Some headers are added to the HTTP request: * `PulsarTopic`: the topic of the record * `PulsarKey`: the key of the record * `PulsarEventTime`: the event time of the record * `PulsarPublishTime`: the publish time of the record * `PulsarMessageId`: the ID of the message contained in the record * `PulsarProperties-*`: each record property is passed with the property name prefixed by `PulsarProperties-` ### Alternatives Creating a separated project for this Sink is rejected since: * this Sink is very useful for developers to test the Pulsar IO framework, transform functions, and to make demos. * the code has a very small footprint with no external dependencies. * it should be visible at the same level as other sinks ### Anything else? _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
