2020-08-13 10:11:46 UTC - moshe baruch: @moshe baruch has joined the channel
----
2020-08-13 12:38:47 UTC - Raman Gupta: @Kirill Kosenko If you do want to keep
schema compat, then another approach is to create a union type of all the
possible topic types.
+1 : Kirill Kosenko
----
2020-08-13 12:45:59 UTC - Raman Gupta: @Shivji Kumar Jha had started some work
to submit a PIP that would improve support for this in Pulsar, however I think
this is one area in which Kafka's design and approach was better than Pulsar's.
Kafka uses the concept of "Subject" to decouple the schema storage and compat
check from the topic.
----
2020-08-13 12:47:35 UTC - Shivji Kumar Jha: Yes @Raman Gupta i still have some
questions from you remaining. I will put those thoughts on pip though so others
can chime in.
+1 : Raman Gupta
----
2020-08-13 13:53:07 UTC - Joshua Decosta: Thanks that worked perfectly.
----
2020-08-13 14:47:47 UTC - Jim M.: @Jim M. has joined the channel
----
2020-08-13 15:08:57 UTC - Ruian: Hi has anyone encountered the `Got exception
java.lang.IllegalAccessError: class
io.airlift.compress.zstd.ZStdRawDecompressor cannot access its superclass
io.airlift.compress.zstd.ZstdFrameDecompressor` error from the pulsar java
client 2.6.0?
----
2020-08-13 15:11:49 UTC - Yarden Arane: @Addison Higham
<https://github.com/apache/pulsar/issues/7814>
----
2020-08-13 15:12:09 UTC - Joshua Decosta: Do proxyRoles need to be explicitly
added to the templates for the helm chart?
----
2020-08-13 15:22:07 UTC - Addison Higham: Yes
----
2020-08-13 15:24:05 UTC - Joshua Decosta: I’m suddenly having issues with my
authorization when deploying on the cluster. My setup works correctly when
using local standalone but I’ve yet to successfully produce or consume a
message even with using a custom admin token. When attempting pulsar-admin
usage it works fine in both situations
----
2020-08-13 15:50:58 UTC - Oliver Isaac: @Oliver Isaac has joined the channel
----
2020-08-13 16:10:08 UTC - Addison Higham: can you give a bit more concrete idea
of how you see the API of that?
Like a function that you pass to the producer that the producer will call as
fast as possible before it fills up it's buffers? I assume the function would
block if it has no messages to send?
That would be a fairly simple extension to the producer and could have some
utility, my biggest concern would just be the producer needing to have a
thread/thread pool. Perhaps a single thread would be sufficient
----
2020-08-13 17:29:27 UTC - Ruian: ```20/08/13 17:21:33 WARN
org.apache.pulsar.client.impl.ClientCnx: [10.140.15.240/10.140.15.240:6650] Got
exception java.lang.IllegalAccessError:
io/airlift/compress/zstd/ZStdRawDecompressor
at
org.apache.pulsar.common.compression.CompressionCodecZstd$1.initialValue(CompressionCodecZstd.java:46)
at
org.apache.pulsar.common.compression.CompressionCodecZstd$1.initialValue(CompressionCodecZstd.java:43)
at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocal.initialize(FastThreadLocal.java:177)
at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocal.get(FastThreadLocal.java:142)
at
org.apache.pulsar.common.compression.CompressionCodecZstd.decode(CompressionCodecZstd.java:90)
at
org.apache.pulsar.client.impl.ConsumerImpl.uncompressPayloadIfNeeded(ConsumerImpl.java:1553)
at
org.apache.pulsar.client.impl.ConsumerImpl.messageReceived(ConsumerImpl.java:1068)
at
org.apache.pulsar.client.impl.ClientCnx.handleMessage(ClientCnx.java:387)
at
org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:182)
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at
org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308)
at
org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422)
at
org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
org.apache.pulsar.shade.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at
org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
at
org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at
org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)```
----
2020-08-13 17:29:28 UTC - Joshua Decosta: I think i know my problem. The proxy
doesn’t seem to be forwarding the authData from the client, so when I’m
checking the authData it’s showing the proxy clients toke but not the clinets
token in the auth data. I was under the impression that the authData would be
passed into the AuthZ methods when checking the client role access.
----
2020-08-13 17:31:44 UTC - Joshua Decosta: So my proxy is just passing along
it’s token in the authData rather than passing along the client token
----
2020-08-13 17:34:00 UTC - Joshua Decosta: Is there something i need to change
in the config to make sure the authData marching up with the role being checked
in the AuthorizationProvider? @Addison Higham
----
2020-08-13 17:35:13 UTC - Addison Higham: `forwardAuthorizationCredentials` in
the proxy settings
----
2020-08-13 17:35:33 UTC - Joshua Decosta: So i have that set to true.
----
2020-08-13 17:36:29 UTC - Joshua Decosta: When i was looking at the original
helm charts i left the block in that disables authZ in proxy and enables
forwardAuthorization
----
2020-08-13 18:37:45 UTC - Joshua Decosta: Do i need to pull client data out of
the authData object or somewhere else?
----
2020-08-13 19:10:40 UTC - Roy Tarantino: Hi all, I'm trying to use reg exp in a
websocket consumer. is this possible? The consumer connects to
<ws://localhost:8080/ws/v2/consumer/persistent/public/default/mytopic.*/11-134-6-141>
----
2020-08-13 19:11:57 UTC - Roy Tarantino: and producer sends to
<ws://localhost:8080/ws/v2/producer/persistent/public/default/mytopictransaction>
----
2020-08-13 19:12:22 UTC - Roy Tarantino: but the consumer never gets the
message. Is this possible with websockets?
----
2020-08-13 20:36:25 UTC - Joshua Decosta: @Addison Higham is there a class I
could look at that might give me a clue as to why the authData being passed to
the broker from the proxy isn’t the client token?
----
2020-08-13 21:32:17 UTC - Addison Higham: @Joshua Decosta just to make sure I
understand, were you previously testing with a proxy? because your auth
provider will get passed both sets of credentials, first the proxy and then the
client
----
2020-08-13 21:32:25 UTC - Addison Higham: (sorry was busy with a few other
things)
----
2020-08-13 21:34:24 UTC - Joshua Decosta: I was originally only testing in
standalone. I’ve since moved to a kubernetes cluster using most of the default
configs and now I’m not seeing the authData for my client token being passed. I
found the check in ProxyConnection for the forwarding but when looking through
the logs and debugging the AuthorizationProvider I’m not seeing my client token
being used as the authData
----
2020-08-13 21:35:10 UTC - Joshua Decosta: Also no need to apologize, I
appreciate the help.
----
2020-08-13 21:35:46 UTC - Addison Higham: in the proxy? or in the broker?
----
2020-08-13 21:36:15 UTC - Joshua Decosta: I have it so authentication occurs in
the proxy and authorization occurs in the broker
----
2020-08-13 21:36:26 UTC - Joshua Decosta: Or at least i think that’s what my
configs are doing
----
2020-08-13 21:36:40 UTC - Addison Higham: just authn not authz right?
----
2020-08-13 21:36:53 UTC - Joshua Decosta: Just authN in proxy
----
2020-08-13 21:37:19 UTC - Joshua Decosta: That is what the forwarding config is
used for right?
----
2020-08-13 21:40:14 UTC - Joshua Decosta: I’m testing all this using the
bin/pulsar-admin/client
----
2020-08-13 21:48:50 UTC - Addison Higham: and just to make sure, you aren't
looking only at
`org.apache.pulsar.broker.authentication.AuthenticationProvider#authenticate`
but you also are looking at
`org.apache.pulsar.broker.authentication.AuthenticationState#authenticate` ?
----
2020-08-13 21:49:27 UTC - Joshua Decosta: I am for sure not looking at that
last method
----
2020-08-13 21:49:49 UTC - Joshua Decosta: What is that method used for?
----
2020-08-13 21:54:52 UTC - Addison Higham: because with the proxy, it doesn't
call the first method. It assumes it is a "mutual auth", which is that the
`AuthenticationState` is for. The
`org.apache.pulsar.broker.authentication.AuthenticationProvider#newAuthState`
has a default implementation that doesn't really check anything.
It probably has the default implementation for backwards compat, but likely,
you always need to override it
----
2020-08-13 21:55:59 UTC - Joshua Decosta: AH, maybe that’s my issue. I was
following along with the defaultProvider and looking into Athenz at some point
and maybe i didn’t realize i needed that method because of that.
----
2020-08-13 21:56:48 UTC - Addison Higham: hrm... on closer inspection I am not
sure how this is supposed to work
----
2020-08-13 21:57:26 UTC - Addison Higham: :face_palm: just realized I was on a
really old Pulsar release to help diagnose a customer issue, let me look at
this agian
----
2020-08-13 21:58:06 UTC - Joshua Decosta: It happens,
----
2020-08-13 22:00:25 UTC - Addison Higham: ah right, are you using a chain of
auth providers and still using the "token" method set by the client?
----
2020-08-13 22:03:31 UTC - Joshua Decosta: The token method from
AuthenticationDataToken? Or are you referencing something from the
AuthenticationToken class?
----
2020-08-13 22:03:57 UTC - Addison Higham: okay, initial impression was right,
you do want to implement that `newAuthState` method as that is also part of the
API used by proxies. The javadocs are fairly bad there as it doesn't make clear
that it is a requirement if you want to use the auth provider in the proxy and
in the broker
----
2020-08-13 22:04:39 UTC - Joshua Decosta: What exactly is that method supposed
to do?
----
2020-08-13 22:04:59 UTC - Joshua Decosta: I see some reference to
onestageauthentication
----
2020-08-13 22:10:13 UTC - Joshua Decosta: I think i see based on
AuthenticationProviderToken. It looks like it’s grabbing some object out of the
body?
----
2020-08-13 22:12:16 UTC - Joshua Decosta: So it looks like i need some
AuthenticationState class?
----
2020-08-13 23:55:12 UTC - Bre Gielissen: @Bre Gielissen has joined the channel
----
2020-08-14 07:00:55 UTC - Rahul Vashishth: I wish to automate performance
testing and display graphical output as a result. I have explored
<http://pulsar.apache.org/docs/en/performance-pulsar-perf/|pulsar-perf> and
<http://openmessaging.cloud/docs/benchmarks/pulsar/|openmessaging benchmark>
for the pulsar.
I am expecting the performance tool to export the
• *client-side metrics* as total msg produced vs consumed, msg loss if any,
throughput, size, test duration, threads, etc.
• *server stats -* memory, cpu, backlog, storage quota used and free
Now the result produced by pulsar-perf is just the latency histogram. While
openmessaging does display a bit more, including publish/consume rate with the
backlog.
Questions -
how do people do performance testing? if there any other tool/framework?
do we have plans to add features in pulsar-perf client to export more metrics,
results than just the latency?
how do people retrieve and display test results?
it seems that openmessaging is not in active development(the last commit was 9
months back, and no one looking for the open PRs), I am not sure if I should
use openmessaging or not. Any suggestions?
@Addison Higham @Sijie Guo
----
2020-08-14 07:15:02 UTC - Sijie Guo: > do we have plans to add features in
pulsar-perf client to export more metrics, results than just the latency?
Feel free to add more features to pulsar-perf. That would benefit a lot of
Pulsar users.
----
2020-08-14 07:22:05 UTC - Rahul Vashishth: @Sijie Guo added some more details
on the original question. How we can get server, client stats for perf
results? I assume people must be doing it.
----
2020-08-14 07:55:33 UTC - Sijie Guo: for server, you can setup Promethues and
Grafana dashboard.
----
2020-08-14 07:55:56 UTC - Sijie Guo: for client stats - sometime people write
their own tools to meet their own requirements.
----
