[GitHub] [pulsar] KMoonSC edited a discussion: How to get properties in org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
GitHub user KMoonSC edited a discussion: How to get properties in
org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
### Version
2.7.1
### What do you want ?
i want to log some information in Broker before messages send to consumer
### What have you done？
i use BrokerInterceptor to do so
here's my BrokerInterceptor:
```
public class LogBrokerInterceptor implements BrokerInterceptor {
private static final Logger logger =
LoggerFactory.getLogger(LogBrokerInterceptor.class);
@Override
public void beforeSendMessage(Subscription subscription, Entry entry,
long[] ackSet, PulsarApi.MessageMetadata msgMetadata) {
List properties = msgMetadata.getPropertiesList();
for (PulsarApi.KeyValue keyValue : properties) {
logger.info("LogBrokerInterceptor beforeSendMessage. properties
key:{} value:{}", keyValue.getKey(), keyValue.getValue());
}
logger.info("LogBrokerInterceptor beforeSendMessage. topic:{}
subscription:{} sequenceId:{} propertiesSize:{}", subscription.getTopic(),
subscription.getName(), msgMetadata.getSequenceId(), properties.size());
}
```
and here's my produce test:
```
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
Producer producer = client.newProducer()
.topic("public/default/my-topic")
.create();
Map properties = new HashMap<>();
properties.put("clientId", "clientId123");
producer.newMessage()
.properties(properties)
.value("data1234".getBytes())
.send();
```
### What did you expect to see?
the properties i settled wound be logged like this:
"LogBrokerInterceptor beforeSendMessage. properties key:clientId
value:clientId123"
### What did you see instead?
there's no such log, but this;
"LogBrokerInterceptor beforeSendMessage. topic:xxx subscription:xxx
sequenceId:xxx propertiesSize:0"
shows that there is no properties in msgMetadata
### Q
so here is the Question:
Am I using the wrong way to get the properties I set to the message ?
If so, how to make it right?
GitHub link: https://github.com/apache/pulsar/discussions/20322
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] KMoonSC edited a discussion: How to get properties in org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
GitHub user KMoonSC edited a discussion: How to get properties in
org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
### Version
2.7.1
### What do you want ?
i want to log some information in Broker before messages send to consumer
### What have you done？
i use BrokerInterceptor to do so
here's my BrokerInterceptor:
`
public class LogBrokerInterceptor implements BrokerInterceptor {
private static final Logger logger =
LoggerFactory.getLogger(LogBrokerInterceptor.class);
@Override
public void beforeSendMessage(Subscription subscription, Entry entry,
long[] ackSet, PulsarApi.MessageMetadata msgMetadata) {
List properties = msgMetadata.getPropertiesList();
for (PulsarApi.KeyValue keyValue : properties) {
logger.info("LogBrokerInterceptor beforeSendMessage. properties
key:{} value:{}", keyValue.getKey(), keyValue.getValue());
}
logger.info("LogBrokerInterceptor beforeSendMessage. topic:{}
subscription:{} sequenceId:{} propertiesSize:{}", subscription.getTopic(),
subscription.getName(), msgMetadata.getSequenceId(), properties.size());
}
`
and here's my produce test:
`
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
Producer producer = client.newProducer()
.topic("public/default/my-topic")
.create();
Map properties = new HashMap<>();
properties.put("clientId", "clientId123");
producer.newMessage()
.properties(properties)
.value("data1234".getBytes())
.send();
`
### What did you expect to see?
the properties i settled wound be logged like this:
"LogBrokerInterceptor beforeSendMessage. properties key:clientId
value:clientId123"
### What did you see instead?
there's no such log, but this;
"LogBrokerInterceptor beforeSendMessage. topic:xxx subscription:xxx
sequenceId:xxx propertiesSize:0"
shows that there is no properties in msgMetadata
### Q
so here is the Question:
Am I using the wrong way to get the properties I set to the message ?
If so, how to make it right?
GitHub link: https://github.com/apache/pulsar/discussions/20322
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] KMoonSC edited a discussion: How to get properties in org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
GitHub user KMoonSC edited a discussion: How to get properties in
org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
### Version
2.7.1
### What do you want ?
i want to log some information in Broker before messages send to consumer
### What have you done？
i use BrokerInterceptor to do so
here's my BrokerInterceptor:
``
public class LogBrokerInterceptor implements BrokerInterceptor {
private static final Logger logger =
LoggerFactory.getLogger(LogBrokerInterceptor.class);
@Override
public void beforeSendMessage(Subscription subscription, Entry entry,
long[] ackSet, PulsarApi.MessageMetadata msgMetadata) {
List properties = msgMetadata.getPropertiesList();
for (PulsarApi.KeyValue keyValue : properties) {
logger.info("LogBrokerInterceptor beforeSendMessage. properties
key:{} value:{}", keyValue.getKey(), keyValue.getValue());
}
logger.info("LogBrokerInterceptor beforeSendMessage. topic:{}
subscription:{} sequenceId:{} propertiesSize:{}", subscription.getTopic(),
subscription.getName(), msgMetadata.getSequenceId(), properties.size());
}
``
and here's my produce test:
`
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
Producer producer = client.newProducer()
.topic("public/default/my-topic")
.create();
Map properties = new HashMap<>();
properties.put("clientId", "clientId123");
producer.newMessage()
.properties(properties)
.value("data1234".getBytes())
.send();
`
### What did you expect to see?
the properties i settled wound be logged like this:
"LogBrokerInterceptor beforeSendMessage. properties key:clientId
value:clientId123"
### What did you see instead?
there's no such log, but this;
"LogBrokerInterceptor beforeSendMessage. topic:xxx subscription:xxx
sequenceId:xxx propertiesSize:0"
shows that there is no properties in msgMetadata
### Q
so here is the Question:
Am I using the wrong way to get the properties I set to the message ?
If so, how to make it right?
GitHub link: https://github.com/apache/pulsar/discussions/20322
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] KMoonSC edited a discussion: How to get properties in org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
GitHub user KMoonSC edited a discussion: How to get properties in
org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
### Version
2.7.1
### What do you want ?
i want to log some information in Broker before messages send to consumer
### What have you done？
i use BrokerInterceptor to do so
here's my BrokerInterceptor:
`
public class LogBrokerInterceptor implements BrokerInterceptor {
private static final Logger logger =
LoggerFactory.getLogger(LogBrokerInterceptor.class);
@Override
public void beforeSendMessage(Subscription subscription, Entry entry,
long[] ackSet, PulsarApi.MessageMetadata msgMetadata) {
List properties = msgMetadata.getPropertiesList();
for (PulsarApi.KeyValue keyValue : properties) {
logger.info("LogBrokerInterceptor beforeSendMessage. properties
key:{} value:{}", keyValue.getKey(), keyValue.getValue());
}
logger.info("LogBrokerInterceptor beforeSendMessage. topic:{}
subscription:{} sequenceId:{} propertiesSize:{}", subscription.getTopic(),
subscription.getName(), msgMetadata.getSequenceId(), properties.size());
}
`
and here's my produce test:
`
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
Producer producer = client.newProducer()
.topic("public/default/my-topic")
.create();
Map properties = new HashMap<>();
properties.put("clientId", "clientId123");
producer.newMessage()
.properties(properties)
.value("data1234".getBytes())
.send();
`
### What did you expect to see?
the properties i settled wound be logged like this:
"LogBrokerInterceptor beforeSendMessage. properties key:clientId
value:clientId123"
### What did you see instead?
there's no such log, but this;
"LogBrokerInterceptor beforeSendMessage. topic:xxx subscription:xxx
sequenceId:xxx propertiesSize:0"
shows that there is no properties in msgMetadata
### Q
so here is the Question:
Am I using the wrong way to get the properties I set to the message ?
If so, how to make it right?
GitHub link: https://github.com/apache/pulsar/discussions/20322
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] aymardika added a comment to the discussion: [configuration between Pulsar and MongoDB Atlas not working]
GitHub user aymardika added a comment to the discussion: [configuration between Pulsar and MongoDB Atlas not working] But how could I maintain the connection permanent ? Please could you help ? Sent from Yahoo Mail on Android On Sun, May 14, 2023 at 12:37 AM, ***@***.***> wrote: The error indicates that you can correctly connect to the MongoDB cluster but "Prematurely reached end of stream". — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***> GitHub link: https://github.com/apache/pulsar/discussions/20318#discussioncomment-5899620 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: [configuration between Pulsar and MongoDB Atlas not working]
GitHub user tisonkun added a comment to the discussion: [configuration between Pulsar and MongoDB Atlas not working] This seems like misconfig on mongodb connect string. No classes from the Pulsar connector occur in the log. I suggest you post a question on MongoDB Altas forum to ask for support on correct config if you're their customer. GitHub link: https://github.com/apache/pulsar/discussions/20318#discussioncomment-5895159 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: [configuration between Pulsar and MongoDB Atlas not working]
GitHub user tisonkun added a comment to the discussion: [configuration between Pulsar and MongoDB Atlas not working] The error indicates that you can correctly connect to the MongoDB cluster but "Prematurely reached end of stream". GitHub link: https://github.com/apache/pulsar/discussions/20318#discussioncomment-5895160 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: [configuration between Pulsar and MongoDB Atlas not working]
GitHub user github-actions[bot] added a comment to the discussion: [configuration between Pulsar and MongoDB Atlas not working] The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/20318#discussioncomment-5895155 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Samreay edited a comment on the discussion: [Python Client] How to subscribe to two topics - both with different schemas?
GitHub user Samreay edited a comment on the discussion: [Python Client] How to
subscribe to two topics - both with different schemas?
What I've come up with doesn't seem optimal, but it works by hiding away
multiple blocking synchronous code blocks into threads and then putting the
messages received into an janus.Queue (asyncio.Queues are not thread safe, so
we swapped to janus). This all works under FastAPI to support the existing REST
endpoints we serve.
```python
class Consumer(Thread):
def __init__(self, queue: janus.Queue, topic: str, schema:
pulsar.schema.AvroSchema):
super().__init__()
self.queue = queue
self.pulsar_client = pulsar.Client("pulsar://pulsar:6650")
self.topic = topic
self.schema = schema
def run(self):
reader = self.pulsar_client.create_reader(
self.topic, start_message_id=pulsar.MessageId.latest,
schema=self.schema
)
while True:
msg = reader.read_next()
topic = msg.topic_name()
if "/" in topic:
topic = topic.split("/")[-1]
message = from_avro(msg.value())
self.queue.sync_q.put_nowait(message)
```
Then when the FastAPI app starts you can kick off the threads, and also launch
an `async` method to get from the `asyncio.Queue` passed into the Consumer
threads.
```python
app = FastAPI()
app.queue = asyncio.Queue()
@app.on_event("startup")
async def startup_event():
consumer_preds = Consumer(app.queue, "predictions", Prediction.schema())
consumer_preds.start()
consumer_trades = Consumer(app.queue, "trades", Trade.schema())
consumer_trades.start()
asyncio.create_task(receive_message())
```
`Prediction.schema` and what not are just some wrappers we have to convert
pydantic to avro schemas.
GitHub link:
https://github.com/apache/pulsar/discussions/20263#discussioncomment-5845219
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Samreay added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas?
GitHub user Samreay added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas? Oh yeah, if we just accept json and do the schema validation ourself, no issues. I was hoping to make use of Pulsars schema registry features and how it enforces things to guarantee a consumer with an incompatible version of the schema can't connect. Figured if the pulsar devs have put in the hard work to implement all of that, I should make use of it! (There's also no rush on our side, the nasty threading approach works, but its very nice. GitHub link: https://github.com/apache/pulsar/discussions/20263#discussioncomment-5880316 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] yaalsn added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas?
GitHub user yaalsn added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas? Not JSON schema, I mean sending JSON string, just bytes. I think this can solve your problem quickly. GitHub link: https://github.com/apache/pulsar/discussions/20263#discussioncomment-5880284 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Samreay added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas?
GitHub user Samreay added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas? Doesn't this have the same issue as Avro, which is if I want to enforce schema on the topics with a well defined producer and consumer, it looks like the synchronous single-schema Consumer in the API means we'd need to turn to multithreading or multiprocessing? Does the JSON schema allow us some way around this? GitHub link: https://github.com/apache/pulsar/discussions/20263#discussioncomment-5880245 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] yaalsn added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas?
GitHub user yaalsn added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas? If pydantic, how about using `JSON` instead of `Avro`, you can convert the models to `JSON` easily https://docs.pydantic.dev/latest/usage/exporting_models/ GitHub link: https://github.com/apache/pulsar/discussions/20263#discussioncomment-5880223 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] HattoriHenzo added a comment to the discussion: Docker image not build with maven
GitHub user HattoriHenzo added a comment to the discussion: Docker image not build with maven It seems to be on Windows. I used this [link](https://stackoverflow.com/questions/27820268/trying-to-run-docker-resulted-in-exit-code-127) to resolve the issue by changing the end of file to **LF** in the file **pulsar\docker\pulsar\scripts\install-pulsar-client.sh**. On VS Code you can use this [link](https://stackoverflow.com/questions/39525417/visual-studio-code-how-to-show-line-endings). GitHub link: https://github.com/apache/pulsar/discussions/20296#discussioncomment-5873123 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd edited a comment on the discussion: Use Automatic Fuzzing to find bugs (e.g. as part of CI / via github action)
GitHub user hpvd edited a comment on the discussion: Use Automatic Fuzzing to find bugs (e.g. as part of CI / via github action) @tisonkun since this was moved from type Issue to Discussion, would be great to remove the automatically set `stale` label and the enhancement label. GitHub link: https://github.com/apache/pulsar/discussions/20252#discussioncomment-5871243 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Support alias for the Schema and Message
GitHub user hpvd added a comment to the discussion: Support alias for the Schema and Message @tisonkun since this was moved from type Issue to Discussion, would be great to remove the automatically set `stale` label. GitHub link: https://github.com/apache/pulsar/discussions/20248#discussioncomment-5871234 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] nlu90 added a comment to the discussion: Pulsar Functions lifecycle and depolyment details.
GitHub user nlu90 added a comment to the discussion: Pulsar Functions lifecycle and depolyment details. If you use `ThreadRuntime`, they all run within the Function Worker process. If you use `ProcessRuntime`, each function's instance gets its own process. If you use `KubernetesRuntime`, each function's instance gets its own pod. GitHub link: https://github.com/apache/pulsar/discussions/20195#discussioncomment-5864425 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Zhen-hao added a comment to the discussion: error new to Pulsar 2.11.0
GitHub user Zhen-hao added a comment to the discussion: error new to Pulsar 2.11.0 I made no progress on this one. It seems that 3.0.0 has corrected this behavior because I don't see it anymore after upgrading to 3.0.0. GitHub link: https://github.com/apache/pulsar/discussions/19805#discussioncomment-5853423 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd edited a comment on the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd edited a comment on the discussion: Build distroless package for better security, smaller size, speed and more sure you are right about false positives! ..but even if the absolute numbers are lower e.g. with a new release of pulsar and an included distro, the mechanism keeps always alive: the more code within a package, the more chances for vulnerabilities (plus the influence on the other 7 points noted in https://github.com/apache/pulsar/discussions/20253#discussion-5173112 ) And if the absolute number of vulnerabilities is low on release day, it will always be higher next week... GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5851978 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd edited a comment on the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd edited a comment on the discussion: Build distroless package for better security, smaller size, speed and more sure you are right about false positives! ..but even if the absolute numbers are lower e.g. with a new release of pulsar and an included distro, the mechanism keeps always alive: the more code within a package, the more chances for vulnerabilities. And if the absolute number of vulnerabilities is low on release day, it will always be higher next week... GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5851978 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more sure you are right about false positives! ..but even if the absolute numbers are lower e.g. with a new release of pulsar and an included distro, the mechanism keeps always alive: the more code within a package, the more chances for vulnerabilities. And if the absolute number is low on release day, it will always be higher next week... GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5851978 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] dave2wave added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user dave2wave added a comment to the discussion: Build distroless package for better security, smaller size, speed and more We should be discussing this with the report for 2.10.4 release since many CVEs have been fixed. I think that this would give us a good basis for comparison. GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5851899 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] dave2wave added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user dave2wave added a comment to the discussion: Build distroless package for better security, smaller size, speed and more About CVEs in a dependency. There can be many types of false positives. 1. The use of the package may already workaround the vulnerability. This happened a lot in projects like POI where XMLBeans was worked around until it was taken out of the Attic and fixed. In the meantime those who did not need Entity Expansion could turn it off. 2. The vulnerable part of the dependency may not be used. GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5851867 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] barryhatfield added a comment to the discussion: LastMessageID, marker Messages and ReplicateSubscriptionState
GitHub user barryhatfield added a comment to the discussion: LastMessageID, marker Messages and ReplicateSubscriptionState + 1 Following GitHub link: https://github.com/apache/pulsar/discussions/20273#discussioncomment-5851647 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] asafm added a comment to the discussion: JVM metrics
GitHub user asafm added a comment to the discussion: JVM metrics I created a new PR for the documentation containing the missing JVM metrics. https://github.com/apache/pulsar-site/pull/567/files GitHub link: https://github.com/apache/pulsar/discussions/20215#discussioncomment-5849088 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more thanks for positive feedback and adding hints for possibilities and classification! -> clicking the up vote button may help to gain further visibility for this topic :-) GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5846869 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: Difference between JSON schema and AVRO schema
GitHub user tisonkun added a comment to the discussion: Difference between JSON schema and AVRO schema cc @codelipenghui @congbobo184 may you have the motivation and use case here? GitHub link: https://github.com/apache/pulsar/discussions/20260#discussioncomment-5845591 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Samreay added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas?
GitHub user Samreay added a comment to the discussion: [Python Client] How to
subscribe to two topics - both with different schemas?
What I've come up with doesn't seem optimal, but it works by hiding away
multiple blocking synchronous code blocks into threads and then putting the
messages received into an asynchio.Queue. This all works under FastAPI to
support the existing REST endpoints we serve.
```python
class Consumer(Thread):
def __init__(self, queue: asyncio.Queue, topic: str, schema:
pulsar.schema.AvroSchema):
super().__init__()
self.queue = queue
self.pulsar_client = pulsar.Client("pulsar://pulsar:6650")
self.topic = topic
self.schema = schema
def run(self):
queue: asyncio.Queue = self.queue
reader = self.pulsar_client.create_reader(
self.topic, start_message_id=pulsar.MessageId.latest,
schema=self.schema
)
while True:
msg = reader.read_next()
topic = msg.topic_name()
if "/" in topic:
topic = topic.split("/")[-1]
message = from_avro(msg.value())
queue.put_nowait(message)
```
Then when the FastAPI app starts you can kick off the threads, and also launch
an `async` method to get from the `asyncio.Queue` passed into the Consumer
threads.
```python
app = FastAPI()
app.queue = asyncio.Queue()
@app.on_event("startup")
async def startup_event():
consumer_preds = Consumer(app.queue, "predictions", Prediction.schema())
consumer_preds.start()
consumer_trades = Consumer(app.queue, "trades", Trade.schema())
consumer_trades.start()
asyncio.create_task(receive_message())
```
`Prediction.schema` and what not are just some wrappers we have to convert
pydantic to avro schemas.
GitHub link:
https://github.com/apache/pulsar/discussions/20263#discussioncomment-5845219
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] lvs071103 edited a discussion: how to delete ledger
GitHub user lvs071103 edited a discussion: how to delete ledger Because the disk space is full, bookkeeper reports the following error, how to delete ledger, or do you have any recommendations ``` 12:04:13.283 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Read for failed on bookie 10.1.69.169:3181 code EIO 12:04:13.283 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Error while reading ledger while reading L6903 E7473 from bookie: 10.1.69.169:3181 12:04:13.306 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Read for failed on bookie 10.1.69.169:3181 code EIO 12:04:13.306 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Error while reading ledger while reading L6903 E7475 from bookie: 10.1.69.169:3181 ``` GitHub link: https://github.com/apache/pulsar/discussions/20266 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user michaeljmarshall added a comment to the discussion: Build distroless package for better security, smaller size, speed and more This would be a great feature to add! GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5844529 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user michaeljmarshall added a comment to the discussion: Build distroless package for better security, smaller size, speed and more Also, pulsar's configuration could be ready for an overhaul. There is an ongoing discussion about configuration here https://lists.apache.org/thread/ok6hnvwsfcckj50471tkfbbhtgo6ng35 GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5844525 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user michaeljmarshall added a comment to the discussion: Build distroless package for better security, smaller size, speed and more > => maybe this is a high value long term topic for Pulsar 4.0 Based on our current versioning scheme, this kind of change could be introduced in a minor release, like 3.1. GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5844519 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Samreay edited a discussion: [Python Client] How to subscribe to two topics - both with different schemas?
GitHub user Samreay edited a discussion: [Python Client] How to subscribe to two topics - both with different schemas? Hi all, I'm trying to write up a simple proof-of-concept application, wherein all it does it subscribe to two separate topics, each with their own different AvroSchema. So far, this doesn't seem possible to do easily with the (synchronous) python client, because each `Consumer` only supports one type of schema, and a `consumer.receive()` blocks. Does anyone know a nice solution for this? GitHub link: https://github.com/apache/pulsar/discussions/20263 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] shodo edited a discussion: Difference between JSON schema and AVRO schema
GitHub user shodo edited a discussion: Difference between JSON schema and AVRO
schema
When checking this section of the doc:
https://pulsar.apache.org/docs/3.0.x/schema-understand/
It's not really clear to me what is the difference between a JSON schema and an
AVRO schema.
When I talk about JSON schema I refer to this specification, basically a JSON
to define how a JSON payload is composed:
http://json-schema.org/understanding-json-schema/index.html
While with avro i refer to this one, a JSON to define how an AVRO payload is
composed:
https://avro.apache.org/docs/
Both the schemas are written in JSON, but their specifications are quite
different.
However If i check the Pulsar doc with example in C++ I see that in the AVRO
example this string is passed:
```C++
static const std::string exampleSchema =
"{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
"\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
Producer producer;
ProducerConfiguration producerConf;
producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
```
while in the JSON example:
```C++
Std::string jsonSchema =
R"({"type":"record","name":"cpx","fields":[{"name":"re","type":"double"},{"name":"im","type":"double"}]})";
SchemaInfo schemaInfo = SchemaInfo(JSON, "JSON", jsonSchema);
```
Although the two strings are instanced in different ways, the content is pretty
similar, and seems they are both respecting the AVRO specification! The only
real difference is that the AVRO one has the "namespace" field that is not
present in the JSON example.
So what's the point of saying that both JSON and AVRO are supported if seems
that in both cases the AVRO specification is used? Am I missing something?
GitHub link: https://github.com/apache/pulsar/discussions/20260
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] shodo edited a discussion: Difference between JSON schema and AVRO schema
GitHub user shodo edited a discussion: Difference between JSON schema and AVRO
schema
When checking this section of the doc:
https://pulsar.apache.org/docs/3.0.x/schema-understand/
It's not really clear to me what is the difference between a JSON schema and an
AVRO schema.
When I talk about JSON schema I refer to this specification, basically a JSON
to define how a JSON payload is composed:
http://json-schema.org/understanding-json-schema/index.html
While with avro i refer to this one, a JSON to define how an AVRO payload is
composed:
https://avro.apache.org/docs/
Both the schemas are written in JSON, but their specifications are quite
different.
However If i check the Pulsar doc with example in C++ I see that in the AVRO
example this string is passed:
```C++
static const std::string exampleSchema =
"{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
"\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
Producer producer;
ProducerConfiguration producerConf;
producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
```
while in the JSON example:
```C++
Std::string jsonSchema =
R"({"type":"record","name":"cpx","fields":[{"name":"re","type":"double"},{"name":"im","type":"double"}]})";
SchemaInfo schemaInfo = SchemaInfo(JSON, "JSON", jsonSchema);
```
Although the two strings are instanced in different ways, the content is pretty
similar, and seems they are both respecting the AVRO specification!
So what's the point of saying that both JSON and AVRO are supported if seems
that in both cases the AVRO specification is used? Am I missing something?
GitHub link: https://github.com/apache/pulsar/discussions/20260
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Pulsar Shell: native image with GraalVM
GitHub user hpvd added a comment to the discussion: Pulsar Shell: native image with GraalVM perfect. thanks! GitHub link: https://github.com/apache/pulsar/discussions/20095#discussioncomment-5838322 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] codelipenghui added a comment to the discussion: Cancel delayed message
GitHub user codelipenghui added a comment to the discussion: Cancel delayed message The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/20255#discussioncomment-5837006 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: Cancel delayed message
GitHub user tisonkun added a comment to the discussion: Cancel delayed message @HarryFQ What do you mean by "cancel"? Do you want the broker to cancel delay messages that are not consumed or the client to do something? GitHub link: https://github.com/apache/pulsar/discussions/20255#discussioncomment-5837015 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] HarryFQ created a discussion: Cancel delayed message
GitHub user HarryFQ created a discussion: Cancel delayed message **Is your feature request related to a problem? Please describe.** Now only add a delayed message above the delayed message, there is no function to cancel the delayed message. We have about 40-50 million delayed messages every day, and currently we are using redis zset to add and cancel. **Describe the solution you'd like** Delayed messages that are not consumed can be cancelled. **Describe alternatives you've considered** A clear and concise description of any alternative solutions or features you've considered. **Additional context** Add any other context or screenshots about the feature request here. GitHub link: https://github.com/apache/pulsar/discussions/20255 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: Pulsar Shell: native image with GraalVM
GitHub user tisonkun added a comment to the discussion: Pulsar Shell: native image with GraalVM Never mind. I manually remove it. GitHub link: https://github.com/apache/pulsar/discussions/20095#discussioncomment-5836748 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more hmm maybe distroless is not the only suitable approach to pay-in on goals named above for distroless coming from related https://github.com/apache/pulsar/discussions/20095#discussioncomment-5836331 Example from the other side: Kafka broker (and Zookeeper) compiled to native using Quarkus and GraalVM https://github.com/ozangunalp/kafka-native GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5836467 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd edited a comment on the discussion: Pulsar Shell: native image with GraalVM
GitHub user hpvd edited a comment on the discussion: Pulsar Shell: native image with GraalVM @tisonkun when making a discussion from an issue, comments in discussion seem not to remove the stale label automatically set before in the issue GitHub link: https://github.com/apache/pulsar/discussions/20095#discussioncomment-5836410 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Pulsar Shell: native image with GraalVM
GitHub user hpvd added a comment to the discussion: Pulsar Shell: native image with GraalVM @tisonkun when making a discussion from an issue, comments in discussion seem not to remove the stale label (automatically set before in the issue) GitHub link: https://github.com/apache/pulsar/discussions/20095#discussioncomment-5836410 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Pulsar Shell: native image with GraalVM
GitHub user hpvd added a comment to the discussion: Pulsar Shell: native image with GraalVM and if this "native" approach is thinkable, it may have a strong relationship to addressing the targets from Build distroless package for better security, smaller size, speed and more #20253 GitHub link: https://github.com/apache/pulsar/discussions/20095#discussioncomment-5836377 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Pulsar Shell: native image with GraalVM
GitHub user hpvd added a comment to the discussion: Pulsar Shell: native image with GraalVM maybe this is not only interesting for Pulsar Shell, but also for other components of pulsar? example from the other side: Kafka broker (and Zookeeper) compiled to native using Quarkus and GraalVM https://github.com/ozangunalp/kafka-native GitHub link: https://github.com/apache/pulsar/discussions/20095#discussioncomment-5836331 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd edited a discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd edited a discussion: Build distroless package for better security, smaller size, speed and more As proofed in practice (e.g. see our own example https://github.com/apache/pulsar/discussions/20253#discussioncomment-5835696 ) there are sometimes (often!) security problems in a container/package whose origin is not the software one build, but in the software which is also situated in this container. In most cases, there is no (little) use case for this additional software. This is where the idea of **distroless containers** comes in and "free" your software: 1) for **better security** 2) fewer bugs 3) smaller packages 4) a faster build process 5) a faster check process (e.g. security scans for CVEs and CWEs) 6) **faster, cheaper and less annoying development** process, because of less noise to understand and fix 7) faster spin-up / faster **dynamic scaling** on load 8) less demanding for needed infrastructure = **less cost** for infrastructure to run on 9) ... Traditional, this approach is somehow strenuous to implement and associated with restrictions. But it looks like **2 new tools makes it pretty easy and straight forward**: **good overview on distroless containers** https://dev.to/dansiviter/distroless-alpine-ci8 and https://blog.chainguard.dev/minimal-container-images-towards-a-more-secure-future/ see last paragraph for how it works **the tools:** source to abk: https://github.com/chainguard-dev/melange abk to oci: https://github.com/chainguard-dev/apko **to debug distroless containers:** official: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/ detail flow: https://iximiuz.com/en/posts/kubernetes-ephemeral-containers/ (this idea/issue was created as follow-up to ongoing progress in distroless for functionmesh see https://github.com/streamnative/function-mesh/issues/448) GitHub link: https://github.com/apache/pulsar/discussions/20253 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd edited a discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd edited a discussion: Build distroless package for better security, smaller size, speed and more As proofed in practice (e.g. see our own example https://github.com/apache/pulsar/discussions/20253#discussioncomment-5835696 ) there are sometimes (often!) security problems in a container/package whose origin is not the software one build, but in the software which is also situated in this container. In most cases, there is no use case for this additional software. This is where the idea of **distroless containers** comes in and "free" your software: 1) for **better security** 2) fewer bugs 3) smaller packages 4) a faster build process 5) a faster check process (e.g. security scans for CVEs and CWEs) 6) **faster, cheaper and less annoying development** process, because of less noise to understand and fix 7) faster spin-up / faster **dynamic scaling** on load 8) less demanding for needed infrastructure = **less cost** for infrastructure to run on 9) ... Traditional, this approach is somehow strenuous to implement and associated with restrictions. But it looks like **2 new tools makes it pretty easy and straight forward**: **good overview on distroless containers** https://dev.to/dansiviter/distroless-alpine-ci8 and https://blog.chainguard.dev/minimal-container-images-towards-a-more-secure-future/ see last paragraph for how it works **the tools:** source to abk: https://github.com/chainguard-dev/melange abk to oci: https://github.com/chainguard-dev/apko **to debug distroless containers:** official: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/ detail flow: https://iximiuz.com/en/posts/kubernetes-ephemeral-containers/ (this idea/issue was created as follow-up to ongoing progress in distroless for functionmesh see https://github.com/streamnative/function-mesh/issues/448) GitHub link: https://github.com/apache/pulsar/discussions/20253 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more for example regarding security, see sources of vulnerabilities in pulsars helm chart. Even an update to latest version of everything would not solve the main problem, that included distros always add a huge part of vulnerabilities => **included distros simply add more code, with more chances for vulnerabilities to be introduced.**  see https://artifacthub.io/packages/helm/apache/pulsar?modal=security-report=vulnerabilities GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5835696 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd edited a discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd edited a discussion: Build distroless package for better security, smaller size, speed and more As proofed in practice there are sometimes (often!) security problems in a container/package whose origin is not the software one build, but in the software which is also situated in this container. In most cases, there is no use case for this additional software. This is where the idea of **distroless containers** comes in and "free" your software: 1) for **better security** 2) fewer bugs 3) smaller packages 4) a faster build process 5) a faster check process (e.g. security scans for CVEs and CWEs) 6) **faster, cheaper and less annoying development** process, because of less noise to understand and fix 7) faster spin-up / faster **dynamic scaling** on load 8) less demanding for needed infrastructure = **less cost** for infrastructure to run on 9) ... Traditional, this approach is somehow strenuous to implement and associated with restrictions. But it looks like **2 new tools makes it pretty easy and straight forward**: **good overview on distroless containers** https://dev.to/dansiviter/distroless-alpine-ci8 and https://blog.chainguard.dev/minimal-container-images-towards-a-more-secure-future/ see last paragraph for how it works **the tools:** source to abk: https://github.com/chainguard-dev/melange abk to oci: https://github.com/chainguard-dev/apko **to debug distroless containers:** official: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/ detail flow: https://iximiuz.com/en/posts/kubernetes-ephemeral-containers/ (this idea/issue was created as follow-up to ongoing progress in distroless for functionmesh see https://github.com/streamnative/function-mesh/issues/448) GitHub link: https://github.com/apache/pulsar/discussions/20253 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd edited a comment on the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd edited a comment on the discussion: Build distroless package for better security, smaller size, speed and more as @michaeljmarshall already said > Note that using a distroless base image is not a viable option for pulsar 2.x > because we use shell scripts to configure each component before executing the > java command and changing that configuration paradigm would be a breaking > change. see https://github.com/apache/pulsar/issues/11269#issuecomment-896325282 => maybe this is a high value long term topic for Pulsar 4.0 GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5835523 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more
GitHub user hpvd added a comment to the discussion: Build distroless package for better security, smaller size, speed and more as @michaeljmarshall already said `Note that using a distroless base image is not a viable option for pulsar 2.x because we use shell scripts to configure each component before executing the java command and changing that configuration paradigm would be a breaking change.` https://github.com/apache/pulsar/issues/11269#issuecomment-896325282 => maybe it's a high value long term topic for Pulsar 4.0 GitHub link: https://github.com/apache/pulsar/discussions/20253#discussioncomment-5835523 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: Use Automatic Fuzzing to find bugs (e.g. as part of CI / via github action)
GitHub user github-actions[bot] added a comment to the discussion: Use Automatic Fuzzing to find bugs (e.g. as part of CI / via github action) The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/20252#discussioncomment-5835376 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] otmanel31 added a comment to the discussion: [question] pulsar broker reboot
GitHub user otmanel31 added a comment to the discussion: [question] pulsar broker reboot Other error later: 10:26:36.465 [ForkJoinPool.commonPool-worker-176] WARN org.apache.pulsar.broker.lookup.TopicLookupBase - Failed to lookup 44a6de43-4e73-42a8-9262-0cc4891c42fd for topic persistent://my_tenant/my_ns/data_down-bf558c9c-a2e6-4961-a627-87229c619abf with error java.util.concurrent.RejectedExecutionException: Thread limit exceeded replacing blocked worker May be as no response from zookeeper, thread continue to increase in broker side ? GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835090 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: [question] pulsar broker reboot
GitHub user github-actions[bot] added a comment to the discussion: [question] pulsar broker reboot The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835091 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] otmanel31 added a comment to the discussion: [question] pulsar broker reboot
GitHub user otmanel31 added a comment to the discussion: [question] pulsar broker reboot sorry, but i didn't find any zookeeper client in my pulsar toolset deployment. Only pulsar-admin, client CLI etc ... let me check if i can install the zkCli.sh. GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835085 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] otmanel31 added a comment to the discussion: [question] pulsar broker reboot
GitHub user otmanel31 added a comment to the discussion: [question] pulsar broker reboot is there another way to get zookeeper data ? from zookeeper logs for example ? GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835086 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] otmanel31 added a comment to the discussion: [question] pulsar broker reboot
GitHub user otmanel31 added a comment to the discussion: [question] pulsar broker reboot Hi, please find below needed datas in files. One file per broker num. we currently have 4 broker: - https://github.com/otmanel31/share/blob/master/pulsar_zookeeper_3_request.json - https://github.com/otmanel31/share/blob/master/requet_broker_0.json - https://github.com/otmanel31/share/blob/master/requet_broker_1.json - https://github.com/otmanel31/share/blob/master/requet_broker_2.json GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835088 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Shoothzj added a comment to the discussion: [question] pulsar broker reboot
GitHub user Shoothzj added a comment to the discussion: [question] pulsar broker reboot via the zookeeper command line client GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835084 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Shoothzj added a comment to the discussion: [question] pulsar broker reboot
GitHub user Shoothzj added a comment to the discussion: [question] pulsar broker reboot This command would work, but before #12102 release, you need to copy a jline jar into `pulsar/lib` directory, it can be easily find in zookeeper binary package or other place. ```bash bin/pulsar zookeeper-shell ``` GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835087 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] otmanel31 added a comment to the discussion: [question] pulsar broker reboot
GitHub user otmanel31 added a comment to the discussion: [question] pulsar broker reboot @Shoothzj , the data is in filesystem of zookeeper under /data ? or via an http request ? or the log content of log file under /data/zookeeper/version-2 ? GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835083 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Shoothzj added a comment to the discussion: [question] pulsar broker reboot
GitHub user Shoothzj added a comment to the discussion: [question] pulsar broker reboot which `LoadSheddingStrategy` are you using? It seems like five broker's load balance doesn't balanced GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835080 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] otmanel31 added a comment to the discussion: [question] pulsar broker reboot
GitHub user otmanel31 added a comment to the discussion: [question] pulsar broker reboot 1) Also, two days ago, we also face an other issue (that trigger a shutdown of our deployement), where brokers timeout on zookeeper call. Before first exception, i catched a lot of logs (Info) in few seconds like below for each topics in all brokers: - 09:49:31.186 [ForkJoinPool.commonPool-worker-1] INFO org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling for persistent://my_tenant/my_ns/action_down-f640fa62-3245-41af-81f5-edf868118a9a 09:49:31.186 [ForkJoinPool.commonPool-worker-1] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://my_tenant/my_ns/action_down-f640fa62-3245-41af-81f5-edf868118a9a] Policies updated successfully For your information, we manage more than 25000 topics, so we have this 2 lines for each active topic. Is there any request to zookeeper when this 2 previous log lines appear ? 2) Then first exception thrown is: 09:50:06.167 [pulsar-ordered-OrderedExecutor-1-0] WARN org.apache.pulsar.broker.service.BrokerService - Got exception when reading persistence policy for persistent://my_tenant/my_ns/data_down-5034936e-c9bb-4720-874d-e7e6e5e6d897: null java.util.concurrent.TimeoutException: null at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) ~[?:1.8.0_252] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[?:1.8.0_252] at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1] at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$34(BrokerService.java:1074) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1] at org.apache.bookkeeper.mledger.util.SafeRun$2.safeRun(SafeRun.java:49) [org.apache.pulsar-managed-ledger-2.6.1.jar:2.6.1] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252] 09:50:14.820 [prometheus-stats-43-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=my_tenant/my_ns java.util.concurrent.TimeoutException: null at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) ~[?:1.8.0_252] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[?:1.8.0_252] at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1] at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:64) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1] at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:1859) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1] at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:97) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1] at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$null$0(NamespaceStatsAggregator.java:65) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:388) ~[org.apache.pulsar-pulsar-common-2.6.1.jar:2.6.1] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:160) ~[org.apache.pulsar-pulsar-common-2.6.1.jar:2.6.1] at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$null$1(NamespaceStatsAggregator.java:64) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:388) ~[org.apache.pulsar-pulsar-common-2.6.1.jar:2.6.1] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:160) ~[org.apache.pulsar-pulsar-common-2.6.1.jar:2.6.1] at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:63) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:388) ~[org.apache.pulsar-pulsar-common-2.6.1.jar:2.6.1] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:160)
[GitHub] [pulsar] Shoothzj added a comment to the discussion: [question] pulsar broker reboot
GitHub user Shoothzj added a comment to the discussion: [question] pulsar
broker reboot
Could U please provider the zookeeper data in path
`/loadbalance/brokers/{broker-num}`
like
GitHub link:
https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835082
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] otmanel31 added a comment to the discussion: [question] pulsar broker reboot
GitHub user otmanel31 added a comment to the discussion: [question] pulsar broker reboot i found loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.OverloadShedder, but no loadSheddingStrategy GitHub link: https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835081 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] abhilashmandaliya created a discussion: Support alias for the Schema and Message
GitHub user abhilashmandaliya created a discussion: Support alias for the Schema and Message Schema and message should support different names/alias while performing SerDe. Like `ObjectMapper` has `@JsonAlias` and `@JsonProperty`. Probably we can use the same feature as Pulsar Schemas use `ObjectMapper` internally. GitHub link: https://github.com/apache/pulsar/discussions/20248 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] eolivelli added a comment to the discussion: Support alias for the Schema and Message
GitHub user eolivelli added a comment to the discussion: Support alias for the Schema and Message I see your problem. but we have to find some way in order to abstract from Jackson Mapper because relying on Jackson Mapper will add an hard dependency for Pulsar. in the case of Avro it is fine to support their annotations because we are supporting Apache Avro. but in this case Pulsar supports JSON in general and not Jackson Mapper, that is a specific library. GitHub link: https://github.com/apache/pulsar/discussions/20248#discussioncomment-5834415 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] codelipenghui added a comment to the discussion: Support alias for the Schema and Message
GitHub user codelipenghui added a comment to the discussion: Support alias for the Schema and Message The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/20248#discussioncomment-5834416 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: Client-Server compatibility matrix
GitHub user tisonkun added a comment to the discussion: Client-Server compatibility matrix cc @RobertIndie @BewareMyPower @momo-jun since you're working on https://pulsar.apache.org/client-feature-matrix/ GitHub link: https://github.com/apache/pulsar/discussions/20109#discussioncomment-5817519 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] raymondBourges added a comment to the discussion: "cannot be cast to class" in a pulsar function producing un new message
GitHub user raymondBourges added a comment to the discussion: "cannot be cast to class" in a pulsar function producing un new message @jiangpengcheng Works perfectly using org.apache.pulsar.client.api.Schema.AVRO(Personne.class) Thanks GitHub link: https://github.com/apache/pulsar/discussions/19962#discussioncomment-5803495 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] pzs-pzs added a comment to the discussion: how consume for backlog message
GitHub user pzs-pzs added a comment to the discussion: how consume for backlog message Maybe you can skip some backlog msg, just read the doc https://pulsar.staged.apache.org/docs/en/admin-api-topics/#skip-messages GitHub link: https://github.com/apache/pulsar/discussions/19723#discussioncomment-5802802 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] jiangpengcheng added a comment to the discussion: "cannot be cast to class" in a pulsar function producing un new message
GitHub user jiangpengcheng added a comment to the discussion: "cannot be cast to class" in a pulsar function producing un new message @raymondBourges seems like you added the `pulsar-client` dependency in your function, you can try with `Schema.Avro(Personne.class)` instead of the `AvroSchema.of(Personne.class)` and remove the `pulsar-client` dependency GitHub link: https://github.com/apache/pulsar/discussions/19962#discussioncomment-5798140 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] AlvaroStream added a comment to the discussion: JVM metrics
GitHub user AlvaroStream added a comment to the discussion: JVM metrics Hello, that is not documented there maybe we can open an issue to include them. But you can see an example [here](https://github.com/apache/pulsar/blob/a22700df665a9ea0a8db98334b6d4337e5e490f3/pulsar-broker/src/test/resources/prometheus_metrics_sample.txt) and you can see the information for [bookeeper](https://github.com/apache/pulsar/blob/a22700df665a9ea0a8db98334b6d4337e5e490f3/grafana/dashboards/bookkeeper.json) of [zookeeper](https://github.com/apache/pulsar/blob/a22700df665a9ea0a8db98334b6d4337e5e490f3/grafana/dashboards/zookeeper.json#L8719) GitHub link: https://github.com/apache/pulsar/discussions/20215#discussioncomment-5789859 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Samreay edited a discussion: How to get the latest message (Python client)?
GitHub user Samreay edited a discussion: How to get the latest message (Python client)? Is there a way to get the latest published message in a topic efficiently using the python client? Right now in our proof of concept we've spun up a redis instance to cache the latest value, but it seems like we're missing something obvious, because I imagine getting the latest published message is a fairly key feature in an event-driven system! I've raised an issue with the Python client over `startMessageInclusive` (https://github.com/apache/pulsar-client-python/issues/115), but I'm aware there are probably many ways to approach things and there might be something simple I've missed. GitHub link: https://github.com/apache/pulsar/discussions/20213 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] niclash added a comment to the discussion: Pulsar Functions lifecycle and depolyment details.
GitHub user niclash added a comment to the discussion: Pulsar Functions lifecycle and depolyment details. Ok, thanks. That helps a lot. 1. Each function getting its own (times the parallelism) OS process? Or can they share OS processes? GitHub link: https://github.com/apache/pulsar/discussions/20195#discussioncomment-5777263 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Samreay added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack
GitHub user Samreay added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack Thanks for the quick response, I appreciate it! Ah `priorityLevel` looks interesting. Is that something only available in the Java client? I can't see it in the [feature matrix](https://docs.google.com/spreadsheets/d/1YHYTkIXR8-Ql103u-IMI18TXLlGStK8uJjDsOOA0T20/edit#gid=1784579914), but a search through the [python docs](https://pulsar.apache.org/api/python/3.1.x/pulsar.Client.html#subscribe) doesn't show up anything about priority. Have I misunderstood something, or is this a feature yet to make it across? > You can completely disable prefetching by setting the consumer receiver queue > to 0: each consumer will only have the message that is currently process and > won't block any other message. That's also good to know, and I see the `client.subscribe` function from above does indeed have a `receiver_queue_size` kwarg to make things simple. I'll give it a whirl this afternoon and see if I can get it all working and report back! GitHub link: https://github.com/apache/pulsar/discussions/20211#discussioncomment-5776889 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] merlimat added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack
GitHub user merlimat added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack > The first thing, which I believe is just not possible due to the nature of > the tech, are priority messages and priority consumers, with failover > subscriptions being a way to get fairly close to a priority consumer. For messages priority, you can use separate topics for each priority. Consumer priority can be set for shared subscriptions: https://pulsar.apache.org/api/client/2.11.x/org/apache/pulsar/client/api/ConsumerBuilder.html#priorityLevel(int) > Is this functionality built into the [shared > subscription](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#shared) > at all, and if not, would anyone have any recommendations about how to > approach the problem? You can completely disable prefetching by setting the consumer receiver queue to 0: each consumer will only have the message that is currently process and won't block any other message. GitHub link: https://github.com/apache/pulsar/discussions/20211#discussioncomment-5776848 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] nlu90 added a comment to the discussion: Pulsar Functions lifecycle and depolyment details.
GitHub user nlu90 added a comment to the discussion: Pulsar Functions lifecycle and depolyment details. 1. Running Function Worker with Pulsar Brokers is fine for testing purposes. In a production env, running Function Worker and Pulsar Broker separately is recommended so your Broker will be more stable. The ThreadRuntime will launch your function inside Function Worker's thread pool, and ProcessRuntime will launch separate JVM processes for your functions. 2.a. You can control how many instances for your function via `pulsar-admin functions create --parallelism `. Once submitted, it won't change during the runtime. 2.b. You are responsible for making your function thread-safe and not blocking on the process method. 3.a. You only need to send the request once. If the same request is submitted multiple times, later requests will be rejected due to the `Already Exists` error. 3.b. Function Worker will try to shut down the instance for 10 seconds timeout gracefully. After 10 seconds, it will forcibly terminate the process. One thing to notice is that, as long as the subscription is not cleaned, the newly updated function instance will start from where the old instances stopped to continue the message processing. GitHub link: https://github.com/apache/pulsar/discussions/20195#discussioncomment-5773367 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] asafm added a comment to the discussion: Pulsar Functions lifecycle and depolyment details.
GitHub user asafm added a comment to the discussion: Pulsar Functions lifecycle and depolyment details. Regarding auto-scaling - there is another scheduler (WorkerService plugin) which runs the function instances on k8s: https://functionmesh.io/docs/. I think it has auto-scale built in. Deploy: Any SQL database for example, including NoSQL like HBase, Cassandra, if you set up a table, it needs to be done once. The same goes for S3 buckets. I'm sure there is a built-in solution in Ansible (pattern) for handling "do it once". GitHub link: https://github.com/apache/pulsar/discussions/20195#discussioncomment-5770097 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] niclash added a comment to the discussion: Pulsar Functions lifecycle and depolyment details.
GitHub user niclash added a comment to the discussion: Pulsar Functions lifecycle and depolyment details. Thanks a lot for taking the time to answer; Pooling; Basically, if there is dynamic creation/destruction of instances during the life-time depending on load. What you write is basically; no it is set up statically by user. Thread-safety; Everyone showcase stateless Functions and having no context it operates in. I don't find myself in that luxurious situation, and to set up the overall/over-arching context, it helps a lot to understand the exact behavior of the underlying framework. I don't really like "don't worry about it", that some systems/frameworks give. Deploy; The thing is, it is a lot simpler for me to let Ansible do the same on plenty of machines, than to do it on one. Pulsar itself sits behind firewall, so I can't reach the Pulsar APIs from my workstation, so Ansible can't execute it on localhost either. Complete; The processing in the function, letting the function return before killing it. And then there is immediately the follow up, what happens if the function has hanged? GitHub link: https://github.com/apache/pulsar/discussions/20195#discussioncomment-5766393 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] asafm added a comment to the discussion: Pulsar Functions lifecycle and depolyment details.
GitHub user asafm added a comment to the discussion: Pulsar Functions lifecycle and depolyment details. I'm not a function expert, but I can contribute a bit of my knowledge in here. > IIUIC, I have the option (among other) to run my functions (trusted) inside > the same JVM as the Pulsar Broker itself, by choosing "Run function workers > with brokers" and the "thread runtime". And if I want to run on the same VM > as the broker, but in separate OS process, I simply follow "process runtime". > Is that correct? I don't need K8s to run functions (I have not set up K8s)? Process runtime will launch a process per function instance. For example, if you specified a function to have 3 instances, 3 process would be launched. Function Worker role (even when run inside Pulsar VM) - you can have many of them. One of them would be the leader and scheduler - meaning it will instruct which of the instances should run each function instance. If this is a production environment, it seems better to separate Function Worker and Broker to separate machines. > Lifecycle of functions is very unclear to me. > a. How many instances are created? > b. One per request? > c. One per key? > d. Are instances pooled? > e. Do the functions need to be thread-safe? > f. Can I control it? > When you deploy or update a function using REST or admin CLI, you can specify how many instances you want per each function. Each instance runs in its own Thread/Process/Pod depending on the runtime. Not per request/key. Not sure about the pooling question. Can you elaborate? Regarding thread safety, I guess only if you share variables, but you shouldn't. > When deploying with "pulsar-admin functions create", do I need to > do that on each broker instance, or just once? What happens if my > Ansible tries to do that in parallel on all instances? > > >From my understanding, Pulsar has a storage area where the function metadata >and the JAR/NAR are stored. You should deploy to Pulsar a function only once. > b. I assume that "pulsar-admin functions update" will let all > functions complete before killing the thread/process. Right? > Compete on what? GitHub link: https://github.com/apache/pulsar/discussions/20195#discussioncomment-5766122 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: "cannot be cast to class" in a pulsar function producing un new message
GitHub user tisonkun added a comment to the discussion: "cannot be cast to class" in a pulsar function producing un new message This seems that the platform classloader is different to function classloader but they both load the `SchemaInfoImpl` class. cc @dlg99 @jiangpengcheng do you have ideas how we can configure the class loading strategy or user code to avoid such exception? GitHub link: https://github.com/apache/pulsar/discussions/19962#discussioncomment-5756151 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] raymondBourges added a comment to the discussion: "cannot be cast to class" in a pulsar function producing un new message
GitHub user raymondBourges added a comment to the discussion: "cannot be cast to class" in a pulsar function producing un new message No one to give me an explanation about the error when the function is called: `Failed to create producer: class org.apache.pulsar.client.impl.schema.SchemaInfoImpl cannot be cast to class org.apache.pulsar.client.impl.schema.SchemaInfoImpl (org.apache.pulsar.client.impl.schema.SchemaInfoImpl is in unnamed module of loader org.apache.pulsar.functions.utils.functioncache.FunctionClassLoaders$ParentFirstClassLoader @2f1ea80d; org.apache.pulsar.client.impl.schema.SchemaInfoImpl is in unnamed module of loader java.net.URLClassLoader @2626b418)` ? Thanks GitHub link: https://github.com/apache/pulsar/discussions/19962#discussioncomment-5755798 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] jeantil added a comment to the discussion: pulsar admin client in pulsar-client-all logs a ClassNotFoundException at startup
GitHub user jeantil added a comment to the discussion: pulsar admin client in pulsar-client-all logs a ClassNotFoundException at startup Hello, Thanks for your answer, I realize I misunderstood and shading is not the root cause. Thanks for clarifying this. I would however argue that this log gives a bad developer experience to users of the pulsar client: - following the normal usage documentation of the client will result in the user's application throwing a full on warning with a `ClassNotFoundException` and a full stack trace. - the warning provides absolutely no indication on how to resolve the issue, not even an indication of why it would be important to use conscrypt instead of the JDK security provider - the pulsar client documentation doesn't mention conscrypt, doesn't explain why users of the client should deploy it So either it is important to use conscrypt, in which case it would be nice to know why or its not and this warning should be downgraded to an INFO log. In both cases, the full stacktrace and classcast exception message doesn't bring much useful information and could be logged at a lower level (probably DEBUG) would you be open to a pull request in to that effect? GitHub link: https://github.com/apache/pulsar/discussions/20057#discussioncomment-5754271 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] BewareMyPower added a comment to the discussion: Using MessageID as a log sequence number downstream? Any better options?
GitHub user BewareMyPower added a comment to the discussion: Using MessageID as a log sequence number downstream? Any better options? > Is it safe to rely on this internal undocumented structure of a MessageID? No. Or at least theoretically not. > log sequence number You can try the `BrokerMetadataInterceptor` feature, which is already used in [KoP](https://github.com/streamnative/kop). ```properties brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor ``` Then you can use the `Message#getIndex` API to get the index, which is similar to Kafka's offset. GitHub link: https://github.com/apache/pulsar/discussions/19969#discussioncomment-5749557 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] niclash created a discussion: Pulsar Functions lifecycle and depolyment details.
GitHub user niclash created a discussion: Pulsar Functions lifecycle and depolyment details. Hi, I have been using Pulsar quite successfully for a bit more than a year now, and quite happy with it. Now, I would like to streamline my app a bit and use Pulsar Functions (on 2.11.x), but I need a little bit of guidance. 1. IIUIC, I have the option (among other) to run my functions (trusted) inside the same JVM as the Pulsar Broker itself, by choosing "Run function workers with brokers" and the "thread runtime". And if I want to run on the same VM as the broker, but in separate OS process, I simply follow "process runtime". Is that correct? I don't need K8s to run functions (I have not set up K8s)? 2. Lifecycle of functions is very unclear to me. a. How many instances are created? b. One per request? c. One per key? d. Are instances pooled? e. Do the functions need to be thread-safe? f. Can I control it? 3. Deployment a. When deploying with "pulsar-admin functions create", do I need to do that on each broker instance, or just once? What happens if my Ansible tries to do that in parallel on all instances? b. I assume that "pulsar-admin functions update" will let all functions complete before killing the thread/process. Right? I guess there will be more questions once I dive deeper. TIA Niclas GitHub link: https://github.com/apache/pulsar/discussions/20195 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] sh-y00 created a discussion: Relationship between Backlog size and Retention size
GitHub user sh-y00 created a discussion: Relationship between Backlog size and Retention size Can someone please tell me the relationship between Backlog size and Retention size, because I don't understand it. ●Assumption Backlog Size：500GB ●Description When I tried to set the Retention size to 100GB, the following error was output. _Retention Quota must exceed configured backlog quota for topic. Please increase retention quota and retry_ _Reason: Retention Quota must exceed configured backlog quota for topic. Please increase retention quota and retry_ ●My understanding Backlog size and Retention size are areas used for different purposes, so there was no dependency between them. ●Question ①I would like to know if there is anything wrong about my understanding. ②I would like to know the relationship between Backlog size and Retention size. GitHub link: https://github.com/apache/pulsar/discussions/20191 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Apurva007 added a comment to the discussion: Bookkeeper Storage recommendations
GitHub user Apurva007 added a comment to the discussion: Bookkeeper Storage recommendations @hangc0276 Great! thanks for helping with the answer! GitHub link: https://github.com/apache/pulsar/discussions/19606#discussioncomment-5738485 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hangc0276 added a comment to the discussion: Bookkeeper Storage recommendations
GitHub user hangc0276 added a comment to the discussion: Bookkeeper Storage recommendations > What is the recommended storage ratio of journal to the ledger disks? We have > 6 ledger SSD SATA disks of 7.5 TB each available. We were wondering how much > journal storage(NVMe) would be required? The journal disk doesn't have the storage size requirement, maybe 100GB is enough. You need to ensure the journal throughput can match with your 6 ledger SSD throughput. > Which file system is recommended for Pulsar? Kafka recommends utilizing xfs > filesystem. Was wondering if its the same for pulsar as well? Both XFS and ext4 can be good options to use with Apache Bookkeeper. If performance is a top priority and you have large segments and high-throughput workloads, XFS may be the better choice. If reliability and stability are more important, then ext4 may be the better choice. GitHub link: https://github.com/apache/pulsar/discussions/19606#discussioncomment-5737468 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hangc0276 edited a comment on the discussion: Bookkeeper Direct Memory Consumption is High
GitHub user hangc0276 edited a comment on the discussion: Bookkeeper Direct Memory Consumption is High Hi @sourabhaggrawal, thank you for raising this issue. I have a few questions. - Which version are you upgrading from? Before upgrading, does the BookKeeper's direct memory issue exist? - BookKeeper 4.14.7 fixed many direct memory leak bugs, would you please upgrade the BookKeeper version to 4.14.7 to see if the direct memory issue still exists? You can upgrade the Pulsar to 2.10.4, which uses BookKeeper 4.14.7 - Would you please share your BookKeeper configuration? GitHub link: https://github.com/apache/pulsar/discussions/19663#discussioncomment-5737373 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] hangc0276 added a comment to the discussion: Bookkeeper Direct Memory Consumption is High
GitHub user hangc0276 added a comment to the discussion: Bookkeeper Direct Memory Consumption is High Hi @sourabhaggrawal, thank you for raising this issue. I have a few questions. - Which version are you upgrading from? Before upgrading, does the BookKeeper's direct memory issue exist? - BookKeeper 4.14.7 fixed many direct memory leak bugs, would you please upgrade the BookKeeper version to 4.14.7 to see if the direct memory issue still exists? You can upgrade the Pulsar to 2.10.4, which uses BookKeeper 4.14.7 - Would you please share you BookKeeper configuration? GitHub link: https://github.com/apache/pulsar/discussions/19663#discussioncomment-5737373 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] startjava edited a comment on the discussion: has MessageListener interface and Transactions DEMO code??
GitHub user startjava edited a comment on the discussion: has MessageListener
interface and Transactions DEMO code??
Consumer side transaction，i has not interface demo：
public class PulsarConsumerTransaction {
public static void main(String[] args) throws Exception {
String serviceUrl =
"pulsar://192.168.23.111:6650,192.168.23.112:6650,192.168.23.113:6650";
PulsarClient pulsarClient =
PulsarClient.builder().serviceUrl(serviceUrl)
.enableTransaction(true)
.build();
String txnTopic1 = "persistent://my-tenant/my-ns/my-txn-topic1";
String txnTopic2 = "persistent://my-tenant/my-ns/my-txn-topic2";
List topicList = new ArrayList<>();
topicList.add(txnTopic1);
topicList.add(txnTopic2);
Transaction transaction = pulsarClient.newTransaction()
.withTransactionTimeout(1, TimeUnit.SECONDS)
.build().get();
Consumer consumer =
pulsarClient.newConsumer(AvroSchema.of(User.class))
.topics(topicList)
.subscriptionName("consume-txn")
.subscriptionType(SubscriptionType.Shared)
.subscribe();
//Messages m = consumer.batchReceive();
int i=0;
while (true) {
Message message = consumer.receive();
try {
// Do something with the message
System.out.println("Message received: " + message.getValue());
// Acknowledge the message so that it can be deleted by the
message broker
consumer.acknowledgeCumulativeAsync(message.getMessageId(),
transaction);
} catch (Exception e) {
// Message failed to process, redeliver later
consumer.negativeAcknowledge(message);
}
i++;
if (i >= 5) {
transaction.commit();
transaction =
pulsarClient.newTransaction().withTransactionTimeout(1,
TimeUnit.SECONDS).build().get();
i = 0;
}
}
}
}
How do I switch to the interface demo code？
GitHub link:
https://github.com/apache/pulsar/discussions/20184#discussioncomment-5726483
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] startjava added a comment to the discussion: has MessageListener interface and Transactions DEMO code??
GitHub user startjava added a comment to the discussion: has MessageListener
interface and Transactions DEMO code??
i has not interface demo：
public class PulsarConsumerTransaction {
public static void main(String[] args) throws Exception {
String serviceUrl =
"pulsar://192.168.23.111:6650,192.168.23.112:6650,192.168.23.113:6650";
PulsarClient pulsarClient =
PulsarClient.builder().serviceUrl(serviceUrl)
.enableTransaction(true)
.build();
String txnTopic1 = "persistent://my-tenant/my-ns/my-txn-topic1";
String txnTopic2 = "persistent://my-tenant/my-ns/my-txn-topic2";
List topicList = new ArrayList<>();
topicList.add(txnTopic1);
topicList.add(txnTopic2);
Transaction transaction = pulsarClient.newTransaction()
.withTransactionTimeout(1, TimeUnit.SECONDS)
.build().get();
Consumer consumer =
pulsarClient.newConsumer(AvroSchema.of(User.class))
// 可以传入多个topic
.topics(topicList)
.subscriptionName("consume-txn")
.subscriptionType(SubscriptionType.Shared)
.subscribe();
//Messages m = consumer.batchReceive();
int i=0;
while (true) {
Message message = consumer.receive();
try {
// Do something with the message
System.out.println("Message received: " + message.getValue());
// Acknowledge the message so that it can be deleted by the
message broker
consumer.acknowledgeCumulativeAsync(message.getMessageId(),
transaction);
} catch (Exception e) {
// Message failed to process, redeliver later
consumer.negativeAcknowledge(message);
}
i++;
if (i >= 5) {
transaction.commit();
transaction =
pulsarClient.newTransaction().withTransactionTimeout(1,
TimeUnit.SECONDS).build().get();
i = 0;
}
}
}
}
How do I switch to the interface demo code？
GitHub link:
https://github.com/apache/pulsar/discussions/20184#discussioncomment-5726483
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] startjava edited a comment on the discussion: has MessageListener interface and Transactions DEMO code??
GitHub user startjava edited a comment on the discussion: has MessageListener
interface and Transactions DEMO code??
i has not interface demo：
public class PulsarConsumerTransaction {
public static void main(String[] args) throws Exception {
String serviceUrl =
"pulsar://192.168.23.111:6650,192.168.23.112:6650,192.168.23.113:6650";
PulsarClient pulsarClient =
PulsarClient.builder().serviceUrl(serviceUrl)
.enableTransaction(true)
.build();
String txnTopic1 = "persistent://my-tenant/my-ns/my-txn-topic1";
String txnTopic2 = "persistent://my-tenant/my-ns/my-txn-topic2";
List topicList = new ArrayList<>();
topicList.add(txnTopic1);
topicList.add(txnTopic2);
Transaction transaction = pulsarClient.newTransaction()
.withTransactionTimeout(1, TimeUnit.SECONDS)
.build().get();
Consumer consumer =
pulsarClient.newConsumer(AvroSchema.of(User.class))
.topics(topicList)
.subscriptionName("consume-txn")
.subscriptionType(SubscriptionType.Shared)
.subscribe();
//Messages m = consumer.batchReceive();
int i=0;
while (true) {
Message message = consumer.receive();
try {
// Do something with the message
System.out.println("Message received: " + message.getValue());
// Acknowledge the message so that it can be deleted by the
message broker
consumer.acknowledgeCumulativeAsync(message.getMessageId(),
transaction);
} catch (Exception e) {
// Message failed to process, redeliver later
consumer.negativeAcknowledge(message);
}
i++;
if (i >= 5) {
transaction.commit();
transaction =
pulsarClient.newTransaction().withTransactionTimeout(1,
TimeUnit.SECONDS).build().get();
i = 0;
}
}
}
}
How do I switch to the interface demo code？
GitHub link:
https://github.com/apache/pulsar/discussions/20184#discussioncomment-5726483
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] 3286360470 added a comment to the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user 3286360470 added a comment to the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 @tisonkun thanks for your reply, your answer perfectly solved my problem! GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5714381 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 This is a known issue https://github.com/apache/pulsar/issues/19918 and has been fixed https://github.com/apache/pulsar/pull/19956 on the master branch. IIRC 3.0 will include this fix. GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5707164 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 This is a known issue https://github.com/apache/pulsar/issues/19918 and has been fixed on the master branch. IIRC 3.0 will include this fix. GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5707164 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 This is a known issue https://github.com/apache/pulsar/issues/20034 and has been fixed on the master branch. IIRC 3.0 will include this fix. GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5707164 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user tisonkun added a comment to the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 This is a known issue and has been fixed on the master branch. IIRC 3.0 will include this fix. GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5707164 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] codelipenghui added a comment to the discussion: [Question]How to use websocket in k8s
GitHub user codelipenghui added a comment to the discussion: [Question]How to use websocket in k8s The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/20171#discussioncomment-5703538 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] MarvinCai added a comment to the discussion: [Question]How to use websocket in k8s
GitHub user MarvinCai added a comment to the discussion: [Question]How to use websocket in k8s @wangyun5583 Can you provide more info? How was the Pulsar cluster deployed on k8s, using the Pulsar Helm chart? If you're asking how to access websocket from outside of the k8s cluster I think you need to update your ingress to add websocket port. GitHub link: https://github.com/apache/pulsar/discussions/20171#discussioncomment-5703536 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
