GitHub user KMoonSC edited a discussion: How to get properties in
org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
### Version
2.7.1
### What do you want ?
i want to log some information in Broker before messages send to consumer
### What have you done？
i use
GitHub user KMoonSC edited a discussion: How to get properties in
org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
### Version
2.7.1
### What do you want ?
i want to log some information in Broker before messages send to consumer
### What have you done？
i use
GitHub user KMoonSC edited a discussion: How to get properties in
org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
### Version
2.7.1
### What do you want ?
i want to log some information in Broker before messages send to consumer
### What have you done？
i use
GitHub user KMoonSC edited a discussion: How to get properties in
org.apache.pulsar.broker.intercept.BrokerInterceptor#beforeSendMessage ?
### Version
2.7.1
### What do you want ?
i want to log some information in Broker before messages send to consumer
### What have you done？
i use
GitHub user aymardika added a comment to the discussion: [configuration between
Pulsar and MongoDB Atlas not working]
But how could I maintain the connection permanent ? Please could you help ?
Sent from Yahoo Mail on Android
On Sun, May 14, 2023 at 12:37 AM, ***@***.***> wrote:
The
GitHub user tisonkun added a comment to the discussion: [configuration between
Pulsar and MongoDB Atlas not working]
This seems like misconfig on mongodb connect string. No classes from the Pulsar
connector occur in the log. I suggest you post a question on MongoDB Altas
forum to ask for
GitHub user tisonkun added a comment to the discussion: [configuration between
Pulsar and MongoDB Atlas not working]
The error indicates that you can correctly connect to the MongoDB cluster but
"Prematurely reached end of stream".
GitHub link:
GitHub user github-actions[bot] added a comment to the discussion:
[configuration between Pulsar and MongoDB Atlas not working]
The issue had no activity for 30 days, mark with Stale label.
GitHub link:
https://github.com/apache/pulsar/discussions/20318#discussioncomment-5895155
This
GitHub user Samreay edited a comment on the discussion: [Python Client] How to
subscribe to two topics - both with different schemas?
What I've come up with doesn't seem optimal, but it works by hiding away
multiple blocking synchronous code blocks into threads and then putting the
messages
GitHub user Samreay added a comment to the discussion: [Python Client] How to
subscribe to two topics - both with different schemas?
Oh yeah, if we just accept json and do the schema validation ourself, no
issues. I was hoping to make use of Pulsars schema registry features and how it
GitHub user yaalsn added a comment to the discussion: [Python Client] How to
subscribe to two topics - both with different schemas?
Not JSON schema, I mean sending JSON string, just bytes. I think this can solve
your problem quickly.
GitHub link:
GitHub user Samreay added a comment to the discussion: [Python Client] How to
subscribe to two topics - both with different schemas?
Doesn't this have the same issue as Avro, which is if I want to enforce schema
on the topics with a well defined producer and consumer, it looks like the
GitHub user yaalsn added a comment to the discussion: [Python Client] How to
subscribe to two topics - both with different schemas?
If pydantic, how about using `JSON` instead of `Avro`, you can convert the
models to `JSON` easily
https://docs.pydantic.dev/latest/usage/exporting_models/
GitHub user HattoriHenzo added a comment to the discussion: Docker image not
build with maven
It seems to be on Windows. I used this
[link](https://stackoverflow.com/questions/27820268/trying-to-run-docker-resulted-in-exit-code-127)
to resolve the issue by changing the end of file to **LF**
GitHub user hpvd edited a comment on the discussion: Use Automatic Fuzzing to
find bugs (e.g. as part of CI / via github action)
@tisonkun since this was moved from type Issue to Discussion, would be great to
remove the automatically set `stale` label and the enhancement label.
GitHub link:
GitHub user hpvd added a comment to the discussion: Support alias for the
Schema and Message
@tisonkun since this was moved from type Issue to Discussion, would be great to
remove the automatically set `stale` label.
GitHub link:
GitHub user nlu90 added a comment to the discussion: Pulsar Functions lifecycle
and depolyment details.
If you use `ThreadRuntime`, they all run within the Function Worker process.
If you use `ProcessRuntime`, each function's instance gets its own process.
If you use `KubernetesRuntime`, each
GitHub user Zhen-hao added a comment to the discussion: error new to Pulsar
2.11.0
I made no progress on this one.
It seems that 3.0.0 has corrected this behavior because I don't see it anymore
after upgrading to 3.0.0.
GitHub link:
GitHub user hpvd edited a comment on the discussion: Build distroless package
for better security, smaller size, speed and more
sure you are right about false positives!
..but even if the absolute numbers are lower e.g. with a new release of pulsar
and an included distro,
the mechanism keeps
GitHub user hpvd edited a comment on the discussion: Build distroless package
for better security, smaller size, speed and more
sure you are right about false positives!
..but even if the absolute numbers are lower e.g. with a new release of pulsar
and an included distro,
the mechanism keeps
GitHub user hpvd added a comment to the discussion: Build distroless package
for better security, smaller size, speed and more
sure you are right about false positives!
..but even if the absolute numbers are lower e.g. with a new release of pulsar
and an included distro,
the mechanism keeps
GitHub user dave2wave added a comment to the discussion: Build distroless
package for better security, smaller size, speed and more
We should be discussing this with the report for 2.10.4 release since many CVEs
have been fixed. I think that this would give us a good basis for comparison.
GitHub user dave2wave added a comment to the discussion: Build distroless
package for better security, smaller size, speed and more
About CVEs in a dependency. There can be many types of false positives.
1. The use of the package may already workaround the vulnerability. This
happened a lot
GitHub user barryhatfield added a comment to the discussion: LastMessageID,
marker Messages and ReplicateSubscriptionState
+ 1
Following
GitHub link:
https://github.com/apache/pulsar/discussions/20273#discussioncomment-5851647
This is an automatically sent email for
GitHub user asafm added a comment to the discussion: JVM metrics
I created a new PR for the documentation containing the missing JVM metrics.
https://github.com/apache/pulsar-site/pull/567/files
GitHub link:
https://github.com/apache/pulsar/discussions/20215#discussioncomment-5849088
GitHub user hpvd added a comment to the discussion: Build distroless package
for better security, smaller size, speed and more
thanks for positive feedback and adding hints for possibilities and
classification!
-> clicking the up vote button may help to gain further visibility for this
GitHub user tisonkun added a comment to the discussion: Difference between JSON
schema and AVRO schema
cc @codelipenghui @congbobo184 may you have the motivation and use case here?
GitHub link:
https://github.com/apache/pulsar/discussions/20260#discussioncomment-5845591
This is an
GitHub user Samreay added a comment to the discussion: [Python Client] How to
subscribe to two topics - both with different schemas?
What I've come up with doesn't seem optimal, but it works by hiding away
multiple blocking synchronous code blocks into threads and then putting the
messages
GitHub user lvs071103 edited a discussion: how to delete ledger
Because the disk space is full, bookkeeper reports the following error, how to
delete ledger, or do you have any recommendations
```
12:04:13.283 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR
GitHub user michaeljmarshall added a comment to the discussion: Build
distroless package for better security, smaller size, speed and more
This would be a great feature to add!
GitHub link:
https://github.com/apache/pulsar/discussions/20253#discussioncomment-5844529
This is an
GitHub user michaeljmarshall added a comment to the discussion: Build
distroless package for better security, smaller size, speed and more
Also, pulsar's configuration could be ready for an overhaul. There is an
ongoing discussion about configuration here
GitHub user michaeljmarshall added a comment to the discussion: Build
distroless package for better security, smaller size, speed and more
> => maybe this is a high value long term topic for Pulsar 4.0
Based on our current versioning scheme, this kind of change could be introduced
in a minor
GitHub user Samreay edited a discussion: [Python Client] How to subscribe to
two topics - both with different schemas?
Hi all, I'm trying to write up a simple proof-of-concept application, wherein
all it does it subscribe to two separate topics, each with their own different
AvroSchema. So
GitHub user shodo edited a discussion: Difference between JSON schema and AVRO
schema
When checking this section of the doc:
https://pulsar.apache.org/docs/3.0.x/schema-understand/
It's not really clear to me what is the difference between a JSON schema and an
AVRO schema.
When I talk
GitHub user shodo edited a discussion: Difference between JSON schema and AVRO
schema
When checking this section of the doc:
https://pulsar.apache.org/docs/3.0.x/schema-understand/
It's not really clear to me what is the difference between a JSON schema and an
AVRO schema.
When I talk
GitHub user hpvd added a comment to the discussion: Pulsar Shell: native image
with GraalVM
perfect. thanks!
GitHub link:
https://github.com/apache/pulsar/discussions/20095#discussioncomment-5838322
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe,
GitHub user codelipenghui added a comment to the discussion: Cancel delayed
message
The issue had no activity for 30 days, mark with Stale label.
GitHub link:
https://github.com/apache/pulsar/discussions/20255#discussioncomment-5837006
This is an automatically sent email for
GitHub user tisonkun added a comment to the discussion: Cancel delayed message
@HarryFQ What do you mean by "cancel"? Do you want the broker to cancel delay
messages that are not consumed or the client to do something?
GitHub link:
GitHub user HarryFQ created a discussion: Cancel delayed message
**Is your feature request related to a problem? Please describe.**
Now only add a delayed message above the delayed message, there is no function
to cancel the delayed message. We have about 40-50 million delayed messages
every
GitHub user tisonkun added a comment to the discussion: Pulsar Shell: native
image with GraalVM
Never mind. I manually remove it.
GitHub link:
https://github.com/apache/pulsar/discussions/20095#discussioncomment-5836748
This is an automatically sent email for commits@pulsar.apache.org.
GitHub user hpvd added a comment to the discussion: Build distroless package
for better security, smaller size, speed and more
hmm maybe distroless is not the only suitable approach to pay-in on goals named
above for distroless
coming from related
GitHub user hpvd edited a comment on the discussion: Pulsar Shell: native image
with GraalVM
@tisonkun when making a discussion from an issue, comments in discussion seem
not to remove the stale label automatically set before in the issue
GitHub link:
GitHub user hpvd added a comment to the discussion: Pulsar Shell: native image
with GraalVM
@tisonkun when making a discussion from an issue, comments in discussion seem
not to remove the stale label (automatically set before in the issue)
GitHub link:
GitHub user hpvd added a comment to the discussion: Pulsar Shell: native image
with GraalVM
and if this "native" approach is thinkable, it may have a strong relationship
to addressing the targets from
Build distroless package for better security, smaller size, speed and more
#20253
GitHub
GitHub user hpvd added a comment to the discussion: Pulsar Shell: native image
with GraalVM
maybe this is not only interesting for Pulsar Shell, but also for other
components of pulsar?
example from the other side:
Kafka broker (and Zookeeper) compiled to native using Quarkus and GraalVM
GitHub user hpvd edited a discussion: Build distroless package for better
security, smaller size, speed and more
As proofed in practice (e.g. see our own example
https://github.com/apache/pulsar/discussions/20253#discussioncomment-5835696 )
there are sometimes (often!) security problems in a
GitHub user hpvd edited a discussion: Build distroless package for better
security, smaller size, speed and more
As proofed in practice (e.g. see our own example
https://github.com/apache/pulsar/discussions/20253#discussioncomment-5835696 )
there are sometimes (often!) security problems in a
GitHub user hpvd added a comment to the discussion: Build distroless package
for better security, smaller size, speed and more
for example regarding security, see sources of vulnerabilities in pulsars helm
chart.
Even an update to latest version of everything would not solve the main
GitHub user hpvd edited a discussion: Build distroless package for better
security, smaller size, speed and more
As proofed in practice
there are sometimes (often!) security problems in a container/package whose
origin is not the software one build, but in the software which is also
situated
GitHub user hpvd edited a comment on the discussion: Build distroless package
for better security, smaller size, speed and more
as @michaeljmarshall already said
> Note that using a distroless base image is not a viable option for pulsar 2.x
> because we use shell scripts to configure each
GitHub user hpvd added a comment to the discussion: Build distroless package
for better security, smaller size, speed and more
as @michaeljmarshall already said
`Note that using a distroless base image is not a viable option for pulsar 2.x
because we use shell scripts to configure each
GitHub user github-actions[bot] added a comment to the discussion: Use
Automatic Fuzzing to find bugs (e.g. as part of CI / via github action)
The issue had no activity for 30 days, mark with Stale label.
GitHub link:
GitHub user otmanel31 added a comment to the discussion: [question] pulsar
broker reboot
Other error later:
10:26:36.465 [ForkJoinPool.commonPool-worker-176] WARN
org.apache.pulsar.broker.lookup.TopicLookupBase - Failed to lookup
44a6de43-4e73-42a8-9262-0cc4891c42fd for topic
GitHub user github-actions[bot] added a comment to the discussion: [question]
pulsar broker reboot
The issue had no activity for 30 days, mark with Stale label.
GitHub link:
https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835091
This is an automatically sent email
GitHub user otmanel31 added a comment to the discussion: [question] pulsar
broker reboot
sorry, but i didn't find any zookeeper client in my pulsar toolset deployment.
Only pulsar-admin, client CLI etc ... let me check if i can install the
zkCli.sh.
GitHub link:
GitHub user otmanel31 added a comment to the discussion: [question] pulsar
broker reboot
is there another way to get zookeeper data ? from zookeeper logs for example ?
GitHub link:
https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835086
This is an automatically
GitHub user otmanel31 added a comment to the discussion: [question] pulsar
broker reboot
Hi, please find below needed datas in files. One file per broker num. we
currently have 4 broker:
- https://github.com/otmanel31/share/blob/master/pulsar_zookeeper_3_request.json
-
GitHub user Shoothzj added a comment to the discussion: [question] pulsar
broker reboot
via the zookeeper command line client
GitHub link:
https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835084
This is an automatically sent email for commits@pulsar.apache.org.
To
GitHub user Shoothzj added a comment to the discussion: [question] pulsar
broker reboot
This command would work, but before #12102 release, you need to copy a jline
jar into `pulsar/lib` directory, it can be easily find in zookeeper binary
package or other place.
```bash
bin/pulsar
GitHub user otmanel31 added a comment to the discussion: [question] pulsar
broker reboot
@Shoothzj , the data is in filesystem of zookeeper under /data ? or via an http
request ? or the log content of log file under /data/zookeeper/version-2 ?
GitHub link:
GitHub user Shoothzj added a comment to the discussion: [question] pulsar
broker reboot
which `LoadSheddingStrategy` are you using? It seems like five broker's load
balance doesn't balanced
GitHub link:
https://github.com/apache/pulsar/discussions/20251#discussioncomment-5835080
This
GitHub user otmanel31 added a comment to the discussion: [question] pulsar
broker reboot
1) Also, two days ago, we also face an other issue (that trigger a shutdown of
our deployement), where brokers timeout on zookeeper call. Before first
exception, i catched a lot of logs (Info) in few
GitHub user Shoothzj added a comment to the discussion: [question] pulsar
broker reboot
Could U please provider the zookeeper data in path
`/loadbalance/brokers/{broker-num}`
like
GitHub user otmanel31 added a comment to the discussion: [question] pulsar
broker reboot
i found
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.OverloadShedder,
but no loadSheddingStrategy
GitHub link:
GitHub user abhilashmandaliya created a discussion: Support alias for the
Schema and Message
Schema and message should support different names/alias while performing SerDe.
Like `ObjectMapper` has `@JsonAlias` and `@JsonProperty`. Probably we can use
the same feature as Pulsar Schemas use
GitHub user eolivelli added a comment to the discussion: Support alias for the
Schema and Message
I see your problem.
but we have to find some way in order to abstract from Jackson Mapper because
relying on Jackson Mapper will add an hard dependency for Pulsar.
in the case of Avro it is
GitHub user codelipenghui added a comment to the discussion: Support alias for
the Schema and Message
The issue had no activity for 30 days, mark with Stale label.
GitHub link:
https://github.com/apache/pulsar/discussions/20248#discussioncomment-5834416
This is an automatically sent
GitHub user tisonkun added a comment to the discussion: Client-Server
compatibility matrix
cc @RobertIndie @BewareMyPower @momo-jun since you're working on
https://pulsar.apache.org/client-feature-matrix/
GitHub link:
GitHub user raymondBourges added a comment to the discussion: "cannot be cast
to class" in a pulsar function producing un new message
@jiangpengcheng
Works perfectly using org.apache.pulsar.client.api.Schema.AVRO(Personne.class)
Thanks
GitHub link:
GitHub user pzs-pzs added a comment to the discussion: how consume for backlog
message
Maybe you can skip some backlog msg, just read the doc
https://pulsar.staged.apache.org/docs/en/admin-api-topics/#skip-messages
GitHub link:
GitHub user jiangpengcheng added a comment to the discussion: "cannot be cast
to class" in a pulsar function producing un new message
@raymondBourges
seems like you added the `pulsar-client` dependency in your function, you can
try with `Schema.Avro(Personne.class)` instead of the
GitHub user AlvaroStream added a comment to the discussion: JVM metrics
Hello,
that is not documented there maybe we can open an issue to include them.
But you can see an example
GitHub user Samreay edited a discussion: How to get the latest message (Python
client)?
Is there a way to get the latest published message in a topic efficiently using
the python client? Right now in our proof of concept we've spun up a redis
instance to cache the latest value, but it seems
GitHub user niclash added a comment to the discussion: Pulsar Functions
lifecycle and depolyment details.
Ok, thanks. That helps a lot.
1. Each function getting its own (times the parallelism) OS process? Or can
they share OS processes?
GitHub link:
GitHub user Samreay added a comment to the discussion: Migrating from RabbitMQ:
shared consumers with new messages only on ack
Thanks for the quick response, I appreciate it!
Ah `priorityLevel` looks interesting. Is that something only available in the
Java client? I can't see it in the
GitHub user merlimat added a comment to the discussion: Migrating from
RabbitMQ: shared consumers with new messages only on ack
> The first thing, which I believe is just not possible due to the nature of
> the tech, are priority messages and priority consumers, with failover
> subscriptions
GitHub user nlu90 added a comment to the discussion: Pulsar Functions lifecycle
and depolyment details.
1. Running Function Worker with Pulsar Brokers is fine for testing purposes. In
a production env, running Function Worker and Pulsar Broker separately is
recommended so your Broker will be
GitHub user asafm added a comment to the discussion: Pulsar Functions lifecycle
and depolyment details.
Regarding auto-scaling - there is another scheduler (WorkerService plugin)
which runs the function instances on k8s: https://functionmesh.io/docs/. I
think it has auto-scale built in.
GitHub user niclash added a comment to the discussion: Pulsar Functions
lifecycle and depolyment details.
Thanks a lot for taking the time to answer;
Pooling; Basically, if there is dynamic creation/destruction of instances
during the life-time depending on load. What you write is basically;
GitHub user asafm added a comment to the discussion: Pulsar Functions lifecycle
and depolyment details.
I'm not a function expert, but I can contribute a bit of my knowledge in here.
> IIUIC, I have the option (among other) to run my functions (trusted) inside
> the same JVM as the Pulsar
GitHub user tisonkun added a comment to the discussion: "cannot be cast to
class" in a pulsar function producing un new message
This seems that the platform classloader is different to function classloader
but they both load the `SchemaInfoImpl` class.
cc @dlg99 @jiangpengcheng do you have
GitHub user raymondBourges added a comment to the discussion: "cannot be cast
to class" in a pulsar function producing un new message
No one to give me an explanation about the error when the function is called:
`Failed to create producer: class
GitHub user jeantil added a comment to the discussion: pulsar admin client in
pulsar-client-all logs a ClassNotFoundException at startup
Hello,
Thanks for your answer, I realize I misunderstood and shading is not the root
cause. Thanks for clarifying this.
I would however argue that this
GitHub user BewareMyPower added a comment to the discussion: Using MessageID as
a log sequence number downstream? Any better options?
> Is it safe to rely on this internal undocumented structure of a MessageID?
No. Or at least theoretically not.
> log sequence number
You can try the
GitHub user niclash created a discussion: Pulsar Functions lifecycle and
depolyment details.
Hi,
I have been using Pulsar quite successfully for a bit more than a year now, and
quite happy with it.
Now, I would like to streamline my app a bit and use Pulsar Functions (on
2.11.x), but I
GitHub user sh-y00 created a discussion: Relationship between Backlog size and
Retention size
Can someone please tell me the relationship between Backlog size and Retention
size, because I don't understand it.
●Assumption
Backlog Size：500GB
●Description
When I tried to set the Retention
GitHub user Apurva007 added a comment to the discussion: Bookkeeper Storage
recommendations
@hangc0276 Great! thanks for helping with the answer!
GitHub link:
https://github.com/apache/pulsar/discussions/19606#discussioncomment-5738485
This is an automatically sent email for
GitHub user hangc0276 added a comment to the discussion: Bookkeeper Storage
recommendations
> What is the recommended storage ratio of journal to the ledger disks? We have
> 6 ledger SSD SATA disks of 7.5 TB each available. We were wondering how much
> journal storage(NVMe) would be
GitHub user hangc0276 edited a comment on the discussion: Bookkeeper Direct
Memory Consumption is High
Hi @sourabhaggrawal, thank you for raising this issue. I have a few questions.
- Which version are you upgrading from? Before upgrading, does the BookKeeper's
direct memory issue exist?
-
GitHub user hangc0276 added a comment to the discussion: Bookkeeper Direct
Memory Consumption is High
Hi @sourabhaggrawal, thank you for raising this issue. I have a few questions.
- Which version are you upgrading from? Before upgrading, does the BookKeeper's
direct memory issue exist?
-
GitHub user startjava edited a comment on the discussion: has MessageListener
interface and Transactions DEMO code??
Consumer side transaction，i has not interface demo：
public class PulsarConsumerTransaction {
public static void main(String[] args) throws Exception {
String
GitHub user startjava added a comment to the discussion: has MessageListener
interface and Transactions DEMO code??
i has not interface demo：
public class PulsarConsumerTransaction {
public static void main(String[] args) throws Exception {
String serviceUrl =
GitHub user startjava edited a comment on the discussion: has MessageListener
interface and Transactions DEMO code??
i has not interface demo：
public class PulsarConsumerTransaction {
public static void main(String[] args) throws Exception {
String serviceUrl =
GitHub user 3286360470 added a comment to the discussion:
Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
@tisonkun thanks for your reply, your answer perfectly solved my problem!
GitHub link:
https://github.com/apache/pulsar/discussions/20175#discussioncomment-5714381
This is an
GitHub user tisonkun edited a comment on the discussion:
Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
This is a known issue https://github.com/apache/pulsar/issues/19918 and has
been fixed https://github.com/apache/pulsar/pull/19956 on the master branch.
IIRC 3.0 will include this fix.
GitHub user tisonkun edited a comment on the discussion:
Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
This is a known issue https://github.com/apache/pulsar/issues/19918 and has
been fixed on the master branch. IIRC 3.0 will include this fix.
GitHub link:
GitHub user tisonkun edited a comment on the discussion:
Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
This is a known issue https://github.com/apache/pulsar/issues/20034 and has
been fixed on the master branch. IIRC 3.0 will include this fix.
GitHub link:
GitHub user tisonkun added a comment to the discussion:
Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
This is a known issue and has been fixed on the master branch. IIRC 3.0 will
include this fix.
GitHub link:
https://github.com/apache/pulsar/discussions/20175#discussioncomment-5707164
GitHub user codelipenghui added a comment to the discussion: [Question]How to
use websocket in k8s
The issue had no activity for 30 days, mark with Stale label.
GitHub link:
https://github.com/apache/pulsar/discussions/20171#discussioncomment-5703538
This is an automatically sent email
GitHub user MarvinCai added a comment to the discussion: [Question]How to use
websocket in k8s
@wangyun5583
Can you provide more info? How was the Pulsar cluster deployed on k8s, using
the Pulsar Helm chart?
If you're asking how to access websocket from outside of the k8s cluster I
think
1 - 100 of 133596 matches
Mail list logo
