2020-01-30 09:14:17 UTC - Hiroyuki Yamada: @Hiroyuki Yamada has joined the
channel
----
2020-01-30 11:29:54 UTC - Matt Collinge: @Matt Collinge has joined the channel
----
2020-01-30 12:55:18 UTC - Alex Yaroslavsky: Thanks a lot!
----
2020-01-30 12:56:27 UTC - Alex Yaroslavsky: Cool, thanks! We are considering
OAuth2 for security.
----
2020-01-30 13:09:57 UTC - Alexandre DUVAL: Will do.
----
2020-01-30 13:20:36 UTC - Roman Popenov: My worker function that is running in
the pod still has issues and doesn’t seem to be registering messages that are
pushed to the topic
----
2020-01-30 13:23:03 UTC - Roman Popenov:
```root@pulsar-mini-roman-bastion-6568485f5b-9fpdj:/pulsar# bin/pulsar-admin
topics stats <persistent://forcepoint-tenant/test-ns/demo-topic-sysmon>
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"averageMsgSize" : 0.0,
"storageSize" : 0,
"backlogSize" : 0,
"publishers" : [ {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"averageMsgSize" : 0.0,
"producerId" : 0,
"metadata" : {
"instance_id" : "0",
"application" : "pulsar-source",
"id" : "forcepoint-tenant/test-ns/s3"
},
"address" : "/10.0.0.254:41086",
"producerName" : "pulsar-mini-roman-1-0",
"connectedSince" : "2020-01-30T13:16:53.382Z",
"clientVersion" : "2.5.0"
} ],
"subscriptions" : {
"my-function-subscription" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"msgBacklog" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"lastExpireTimestamp" : 0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "9537e",
"availablePermits" : 1000,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : {
"instance_id" : "0",
"application" : "pulsar-function",
"id" : "forcepoint-tenant/test-ns/etl-function"
},
"address" : "/10.0.1.124:45440",
"connectedSince" : "2020-01-30T13:17:23.857Z",
"clientVersion" : "2.5.0"
} ],
"isReplicated" : false
}
},
"replication" : { },
"deduplicationStatus" : "Disabled",
"bytesInCounter" : 6856177,
"msgInCounter" : 2
}```
----
2020-01-30 13:23:13 UTC - Roman Popenov: It is subscribed to it
----
2020-01-30 13:24:11 UTC - Roman Popenov:
```root@pulsar-mini-roman-bastion-6568485f5b-9fpdj:/pulsar#
/pulsar/bin/pulsar-admin functions create --jar
/tmp/Pulsar-Demo-SKO-1.0-SNAPSHOT-uberjar.jar --classname
com.forcepoint.gemini.pulsar.functions.PulsarSysmonTopicETL --namespace test-ns
--tenant forcepoint-tenant --inputs
<persistent://forcepoint-tenant/test-ns/demo-topic-sysmon> --output
<persistent://forcepoint-tenant/test-ns/demo-topic-sysmon-cim> --subs-name
my-function-subscription --cpu 1 --ram 3073741824 --disk 10737418240
--parallelism 1 --name etl-function
"Created successfully"```
It was created and the pod is running:
```pf-forcepoint-tenant-test-ns-etl-function-0 1/1 Running 0
6m55s```
----
2020-01-30 13:25:54 UTC - Roman Popenov: I do see the following in the broker
logs:
```13:25:06.393 [bookkeeper-io-14-1] WARN
org.apache.bookkeeper.proto.PerChannelBookieClient - Exception caught on:[id:
0x746d95a0, L:/10.0.1.224:40326 ! R:/10.0.0.242:3181] cause:
<http://java.io|java.io>.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_232]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
~[?:1.8.0_232]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
~[?:1.8.0_232]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_232]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
~[?:1.8.0_232]
at
io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:408)
~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:931)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:361)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:694)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
13:25:06.394 [bookkeeper-io-14-1] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x746d95a0, L:/10.0.1.224:40326 ! R:/10.0.0.242:3181]
13:25:06.394 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN
org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (6, 0):
Bookie handle is not available
13:25:06.394 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN
org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (6, 1):
Bookie handle is not available
13:25:06.396 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN
org.apache.bookkeeper.client.BookieWatcherImpl - replaceBookie for bookie:
10.0.0.242:3181 in ensemble: [10.0.0.242:3181, 10.0.0.224:3181] is not adhering
to placement policy and chose 10.0.1.125:3181. excludedBookies
[10.0.0.242:3181, 10.0.0.225:3181] and quarantinedBookies []```
----
2020-01-30 13:29:33 UTC - Roman Popenov: And this is the stats of the source:
```13:17:50.659 [pulsar-timer-4-1] INFO
com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider
initialized
13:17:53.066 [pulsar-timer-4-1] INFO
org.apache.pulsar.client.impl.ProducerStatsRecorderImpl -
[<persistent://forcepoint-tenant/test-ns/demo-topic-sysmon>]
[pulsar-mini-roman-1-0] Pending messages: 2 --- Publish throughput: 0.03 msg/s
--- 6.90 Mbit/s --- Latency: med: 0.000 ms - 95pct: 0.000 ms - 99pct: 0.000 ms
- 99.9pct: 0.000 ms - max: -∞ ms --- Ack received rate: 0.00 ack/s --- Failed
messages: 0
13:28:53.081 [pulsar-timer-4-1] INFO
org.apache.pulsar.client.impl.ProducerStatsRecorderImpl -
[<persistent://forcepoint-tenant/test-ns/demo-topic-sysmon>]
[pulsar-mini-roman-1-0] Pending messages: 4 --- Publish throughput: 0.03 msg/s
--- 6.90 Mbit/s --- Latency: med: 0.000 ms - 95pct: 0.000 ms - 99pct: 0.000 ms
- 99.9pct: 0.000 ms - max: -∞ ms --- Ack received rate: 0.00 ack/s --- Failed
messages: 0```
----
2020-01-30 13:29:47 UTC - Roman Popenov: So the messages are pending
----
2020-01-30 13:30:00 UTC - Roman Popenov: Not sure what is going on :sweat_smile:
----
2020-01-30 13:34:08 UTC - Roman Popenov: Oh, this might explain it:
```13:30:42.559 [bookie-io-1-1] ERROR
org.apache.bookkeeper.proto.BookieRequestHandler - Unhandled exception occurred
in I/O thread or handler
io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds
5242880: 5313378 - discarded
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513)
~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:491)
~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:378)
~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:421)
~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:334)
~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)
~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]```
----
2020-01-30 13:34:21 UTC - Roman Popenov: `TooLongFrameException`
----
2020-01-30 13:50:22 UTC - Roman Popenov: This wasn’t always the behavior, was
it?
----
2020-01-30 14:07:34 UTC - Roman Popenov: Is there a place where it can be
configured now?
----
2020-01-30 14:33:58 UTC - Alexandre DUVAL: Hi, topic compaction creates a
compacted copy of the current topic? I read that the "history" of a topic is
kept even after compaction somewhere IIRC, but the cookbook on compaction
doesnt mention it.
----
2020-01-30 16:01:45 UTC - David Kjerrumgaard: @Roman Popenov It looks like
there is an open issue on this. <https://github.com/apache/pulsar/issues/3832>
----
2020-01-30 16:04:11 UTC - Roman Popenov: I didn’t setup any TLS
----
2020-01-30 16:06:50 UTC - David Kjerrumgaard: You could use OAuth2 JWT for
authentication, but that poses an issue with token expiration and being able to
revoke a token that has been compromised, etc. Whereas TLS client certs are
easier to revoke.
----
2020-01-30 16:10:07 UTC - Roman Popenov: Also, the URLs are:
```{
"serviceUrl" :
"<http://pulsar-mini-roman-broker.pulsar.svc.cluster.local:8080/>",
"brokerServiceUrl" :
"<pulsar://pulsar-mini-roman-broker.pulsar.svc.cluster.local:6650/>"
}```
----
2020-01-30 16:32:54 UTC - Roman Popenov: I can confirm this has to do with the
message size
----
2020-01-30 16:33:40 UTC - Roman Popenov: The setting for messages larger than 5
mbs is not picked up anymore
----
2020-01-30 16:33:50 UTC - Roman Popenov: Is the setting for messages larger
than 5 MBs not picked up anymore?
----
2020-01-30 16:34:04 UTC - David Kjerrumgaard: @Roman Popenov My guess is that
in the case where TLS was enabled (on the JIRA), that the encryption caused
the message size to exceed the 5MB limit and fail.
----
2020-01-30 16:34:35 UTC - Roman Popenov: I don’t have TLS enabled
----
2020-01-30 16:34:48 UTC - Roman Popenov: nor any auth or encryption
----
2020-01-30 16:43:18 UTC - David Kjerrumgaard: Based on your error message, it
was picked up. You just sent a message that exceeded that threshold
----
2020-01-30 16:44:12 UTC - David Kjerrumgaard: Adjusted frame length exceeds
5242880 <---- Looks like 5MB
----
2020-01-30 16:44:45 UTC - Alex Yaroslavsky: We already have an automatic JWT
renewal mechanism in place.
----
2020-01-30 16:45:18 UTC - Alex Yaroslavsky: And it seems that Pulsar doesn't
recheck CRL after a connection was already established
(<https://github.com/apache/pulsar/wiki/PIP-20:-Mechanism-to-revoke-TLS-authentication>)
----
2020-01-30 16:45:47 UTC - Roman Popenov: I have previously set
```maxMessageSize: "62428800"```
in broker config, and the borker did pick it up
----
2020-01-30 16:46:01 UTC - Roman Popenov: I’ve also tried 62428800 and 52428800
----
2020-01-30 16:46:30 UTC - Roman Popenov: And it is picked up, but the exception
is always the same:
```Adjusted frame length exceeds 5242880: 5313378```
----
2020-01-30 16:46:43 UTC - Roman Popenov: This happens if the message exceeds
5242880 bytes
----
2020-01-30 16:46:54 UTC - Roman Popenov: basically any message over 5 MBs
----
2020-01-30 16:52:08 UTC - Roman Popenov: Could it be related to
----
2020-01-30 16:52:09 UTC - Roman Popenov:
<https://github.com/apache/pulsar/issues/4708>
----
2020-01-30 17:00:33 UTC - Roman Popenov: In code, it’s still set as
```bkConf.setNettyMaxFrameSizeBytes(conf.getMaxMessageSize() +
Commands.MESSAGE_SIZE_FRAME_PADDING);```
----
2020-01-30 17:02:34 UTC - Alexandre DUVAL: @Matteo Merli
----
2020-01-30 17:16:18 UTC - David Kjerrumgaard: Is it possible to decrease the
message size? Chunk the message into smaller pieces?
----
2020-01-30 17:16:42 UTC - Roman Popenov: Yeah, that’s what I will have to do or
use 2.4.2, it used to work in that release
----
2020-01-30 17:17:20 UTC - Konstantinos Papalias: Do we still need
AggresiveOpts, they have been completely removed from JDK13 now,
<https://dzone.com/articles/jdk-13-what-is-aggressiveopts>
```apache-pulsar-2.5.0 % bin/pulsar standalone
Unrecognized VM option 'AggressiveOpts'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.```
----
2020-01-30 17:17:46 UTC - David Kjerrumgaard: Yea, that was my suspicion. This
is will need to be changed to support larger message sizes.
----
2020-01-30 17:19:11 UTC - Roman Popenov: I assumed that the message size
followed the
```maxMessageSize```
----
2020-01-30 17:19:25 UTC - Roman Popenov: and that the netty size also followed
this + padding
----
2020-01-30 17:55:49 UTC - Roman Popenov: It’s really weird, the settings have
picked up now, and it does work with larger messages. I will try to repro the
issue
thinking_face : David Kjerrumgaard
----
2020-01-30 18:10:38 UTC - Ruslan Sheremet: @Ruslan Sheremet has joined the
channel
----
2020-01-30 18:55:12 UTC - Ruslan Sheremet: Hi everyone,
This feature <https://pulsar.apache.org/docs/en/security-jwt/> working in
`standalone` mode? I have tried this using `client.conf`, `broker.conf` and
`broker.conf` configurations, but auth still not required when I use CLI or a
nodejs client. Server run without errors and all working fine, but without
enabling Auth via JWT.
`init.sh` used in `docker-compose.yaml` :
```#!/bin/bash
set -e
bin/apply-config-from-env.py conf/broker.conf conf/proxy.conf
conf/client.conf \
&& bin/pulsar standalone -a pulsar```
`docker-compose.yaml` :
```version: '3'
services:
pulsar:
image: apachepulsar/pulsar-all
command: bash init.sh
environment:
PULSAR_HOSTNAME: pulsar
PULSAR_PREFIX_webServiceUrl: <http://pulsar:8080/>
PULSAR_PREFIX_brokerServiceUrl: <pulsar://pulsar:6650/>
PULSAR_PREFIX_brokerWebServiceURL: <http://pulsar:8080/>
PULSAR_PREFIX_brokerServiceURL: <pulsar://pulsar:6650/>
PULSAR_PREFIX_advertisedAddress: pulsar
PULSAR_PREFIX_clusterName: standalone
PULSAR_PREFIX_authorizationEnabled: "true"
PULSAR_PREFIX_authenticationEnabled: "true"
PULSAR_PREFIX_authPlugin:
org.apache.pulsar.client.impl.auth.AuthenticationToken
PULSAR_PREFIX_authParams:
token:eyJhbGciOiJSUzI1NiJ9:eyJzdWIiOiJwdWxzYXIifQ:H0BbDDvywyrept3IouUAzA7-frMBfc-kMa0Bo3JAhiMe-tEhy5QP8Ds3_KVJvyTaE0YI-l5eBK2xucJ3XEdN0x4yMyuXP_VB50mqFoaaLnLL3mpa0qA9kt1sHenmjvijW3yOYs0M8K616Hj_x4hqQq3l-oJ3SKR049LHNm_SvDo478lxo4fx-xqlErRWz7-BJjnYpB0avVEs_-MSaST2PEtAso7hr4XpA84Eh7ydDLCotk3Hm89k9o-e-n6n-Uk-4nwY-PpYyPasHsC7xyqAILvAsLdYeFa3IHDAbnUfvRtixa_phcElM3IU893F-fK1b1po2DKGFAgRW-ie3loLBw
PULSAR_PREFIX_authenticationProviders:
org.apache.pulsar.broker.authentication.AuthenticationProviderToken
PULSAR_PREFIX_proxyRoles: proxy,pulsar,pulsar-pulsar
PULSAR_PREFIX_superUserRoles: admin,pulsar,proxy,pulsar-pulsar
PULSAR_PREFIX_tokenPublicKey: file:////pulsar/pulsar-public.key
PULSAR_PREFIX_brokerClientAuthenticationPlugin:
org.apache.pulsar.client.impl.auth.AuthenticationToken
PULSAR_PREFIX_brokerClientAuthenticationParameters:
token:eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJwdWxzYXIifQ.H0BbDDvywyrept3IouUAzA7-frMBfc-kMa0Bo3JAhiMe-tEhy5QP8Ds3_KVJvyTaE0YI-l5eBK2xucJ3XEdN0x4yMyuXP_VB50mqFoaaLnLL3mpa0qA9kt1sHenmjvijW3yOYs0M8K616Hj_x4hqQq3l-oJ3SKR049LHNm_SvDo478lxo4fx-xqlErRWz7-BJjnYpB0avVEs_-MSaST2PEtAso7hr4XpA84Eh7ydDLCotk3Hm89k9o-e-n6n-Uk-4nwY-PpYyPasHsC7xyqAILvAsLdYeFa3IHDAbnUfvRtixa_phcElM3IU893F-fK1b1po2DKGFAgRW-ie3loLBw
volumes:
- /data/servers/pulsar:/pulsar/data
- /data/servers/pulsar-public.key:/pulsar/pulsar-public.key
- ./pulsar/init.sh:/pulsar/init.sh
ports:
- 6650:6650
- 10180:8080
- 18443:8443
- 6651:6651```
`/data/servers/pulsar-public.key` successfully placed to `/pulsar/*`
directory in container and I think should be readable by
`file:////pulsar/pulsar-public.key` (absolute path).
Also I have tried to overwrite config files in containers via volumes mount to
files, but result the same as with env vars.
----
2020-01-30 18:58:38 UTC - ravi satya durga prasad Yenugula: Thanks the problem
was in the document command not sure what the issue one thing is it has `$` I
just made it as a single line command and it worked
```docker run -it -p 6650:6650 -p 8080:8080 --mount
source=pulsardata,target=/pulsar/data --mount
source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.5.0 bin/pulsar
standalone```
----
2020-01-30 19:08:25 UTC - David Kjerrumgaard: @Ruslan Sheremet I have it
working in a docker image I am using for my book. You can find the code here.
<https://github.com/david-streamlio/pulsar-in-action/tree/master/docker-images/pulsar-standalone>
+1 : Ruslan Sheremet
----
2020-01-30 19:09:08 UTC - David Kjerrumgaard: It should give you a solid
example of the steps required. Pay particular attention to the configs in the
conf/standalone.conf file
----
2020-01-30 19:09:57 UTC - ravi satya durga prasad Yenugula: Hi, I ran the
docker command
```docker run -it -p 6650:6650 -p 8080:8080 --mount
source=pulsardata,target=/pulsar/data --mount
source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.5.0 bin/pulsar
standalone```
and trying to access the web UI then I am getting this error
----
2020-01-30 19:11:46 UTC - ravi satya durga prasad Yenugula: Any Idea ?
----
2020-01-30 19:12:19 UTC - Roman Popenov: UI is on port 8080?
----
2020-01-30 19:12:54 UTC - Roman Popenov: I thought that would be the port for
admin commands
----
2020-01-30 19:13:04 UTC - Roman Popenov: that you can manage the cluster
through REST requests
----
2020-01-30 19:13:39 UTC - ravi satya durga prasad Yenugula: I am new not sure
about the tool setup and its functionality
----
2020-01-30 19:14:05 UTC - Roman Popenov:
<https://github.com/apache/pulsar-manager> - this manager is running on port
9527
----
2020-01-30 19:14:13 UTC - Roman Popenov: I am not sure if it is running in
standalone
----
2020-01-30 19:14:37 UTC - ravi satya durga prasad Yenugula: Doc says
`<http://localhost:8080>` so I was try it
----
2020-01-30 19:14:38 UTC - Roman Popenov: ```docker pull
apachepulsar/pulsar-manager:v0.1.0
docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e
REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e
URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e
PASSWORD=pulsar -e LOG_LEVEL=DEBUG -v $PWD:/data
apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
----
2020-01-30 19:16:16 UTC - Ruslan Sheremet: You should use 9527 port in browser
----
2020-01-30 19:16:29 UTC - ravi satya durga prasad Yenugula: I am using
`standalone` in the command they mentioned `-p 6650:6650 -p 8080:8080` so
its should ne either one. Doc saw 8080 is the http port I am assuming it will
have a UI and the 6650 is a pulsar client port
----
2020-01-30 19:17:42 UTC - ravi satya durga prasad Yenugula: `9527` is not
working and I am running just the standalone version
----
2020-01-30 19:18:07 UTC - Roman Popenov: No, the port 8080 is used for admin
commands and talk to the cluster via rest api
----
2020-01-30 19:18:28 UTC - Roman Popenov: The closest to the UI management of
the cluster is
<https://pulsar.apache.org/docs/en/administration-pulsar-manager/>
----
2020-01-30 19:18:38 UTC - ravi satya durga prasad Yenugula: Ok
----
2020-01-30 19:18:43 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula I
mean pulsar-manager:v0.1.0 with apachepulsar/pulsar
----
2020-01-30 19:18:50 UTC - ravi satya durga prasad Yenugula: so it will not have
a UI
----
2020-01-30 19:18:57 UTC - Roman Popenov: For the Admin API:
<https://pulsar.apache.org/docs/en/admin-api-overview/>
----
2020-01-30 19:19:38 UTC - Ruslan Sheremet: @Roman Popenov Also you should edit
nginx config and use domain or IP of your pulsar server
----
2020-01-30 19:20:38 UTC - Ruslan Sheremet: @Roman Popenov
```server {
listen 9527;
server_name 0.0.0.0;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
location /admin {
proxy_pass <http://pulsar:8080>;
}
location /pulsar-admin-doc {
proxy_pass <http://pulsar-admin-swagger:8080>;
}
location /pulsar-manager {
proxy_pass <http://localhost:7750>;
}
location /lookup {
proxy_pass <http://localhost:7750>;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}```
----
2020-01-30 19:24:09 UTC - Roman Popenov: So the suggested command doesn’t work?
----
2020-01-30 19:24:17 UTC - Roman Popenov:
<https://pulsar.apache.org/docs/en/administration-pulsar-manager/>
----
2020-01-30 19:24:37 UTC - ravi satya durga prasad Yenugula: I am running the
commands
----
2020-01-30 19:26:37 UTC - ravi satya durga prasad Yenugula:
----
2020-01-30 19:26:45 UTC - ravi satya durga prasad Yenugula: docker run is
throwing error
----
2020-01-30 19:26:47 UTC - Ruslan Sheremet: @Roman Popenov It is working in case
if you are run apachepulsar/pulsar-manager and apachepulsar/pulsar on
localhost, but if you want run in VM, this configuration should be changed via
nginx config: `/etc/nginx/conf.d/default.conf` (path in Docker container of
the apachepulsar/pulsar-manager)
----
2020-01-30 19:28:24 UTC - Roman Popenov: Thanks! :thanks:
----
2020-01-30 19:28:28 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula
$PWD means current dir
----
2020-01-30 19:28:36 UTC - Roman Popenov: Yes
----
2020-01-30 19:29:18 UTC - Ruslan Sheremet: @Roman Popenov Please note
``` location /pulsar-admin-doc {
proxy_pass <http://pulsar-admin-swagger:8080>;
}```
this is my another config for Swagger API
----
2020-01-30 19:29:45 UTC - Ruslan Sheremet: @Roman Popenov You can exclude this
from config
heavy_check_mark : Roman Popenov
----
2020-01-30 19:29:53 UTC - ravi satya durga prasad Yenugula: so show I replace
it. In Windows power shell `$PWD` will work
----
2020-01-30 19:42:33 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula On
8080 available only Pulsar API with CORS origin. For UI Manager please use
Pulsar Manager, default port is `9527`
----
2020-01-30 19:49:01 UTC - ravi satya durga prasad Yenugula: I ran this
```docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e
REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e
URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e
PASSWORD=pulsar -e LOG_LEVEL=DEBUG -manager:v0.1.0 /bin/sh```
Error: invalid argument "anager:v0.1.0" for "-m, --memory" flag: invalid size:
'anager:v0.1.0'
then this
```docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e
REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e
URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e
PASSWORD=pulsar -e LOG_LEVEL=DEBUG```
error: "docker run" requires at least 1 argument.
See 'docker run --help'.
Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
Run a command in a new container
----
2020-01-30 19:52:38 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula
You are miss apachepulsar/pulsar-manager:v0.1.0 /bin/sh at the end of run
command
----
2020-01-30 19:55:02 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula
```docker run -it -p 9527:9527 \
-e REDIRECT_HOST=<http://192.168.0.104> \
-e REDIRECT_PORT=9527 \
-e DRIVER_CLASS_NAME=org.postgresql.Driver \
-e URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' \
-e USERNAME=pulsar \
-e PASSWORD=pulsar \
-e LOG_LEVEL=DEBUG \
apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
----
2020-01-30 20:02:42 UTC - ravi satya durga prasad Yenugula: This is running
```docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e
REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e
URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e
PASSWORD=pulsar -e LOG_LEVEL=DEBUG apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
just changed it to single line
----
2020-01-30 20:24:19 UTC - ravi satya durga prasad Yenugula:
----
2020-01-30 20:24:52 UTC - ravi satya durga prasad Yenugula: When login its is
throwing few errors what are those?
```<!doctype html><html lang="en"><head><title>HTTP Status
500 – Internal Server Error</title><style type="text/css">h1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
h2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
h3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}
a {color:black;} a.name {color:black;} .line
{height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
Status 500 – Internal Server Error</h1></body></html>```
----
2020-01-30 20:38:42 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula
Please follow all changes required for `nginx` config from start of this thread
----
2020-01-30 20:41:50 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula
You should open
```<http://192.168.0.104:9527>```
----
2020-01-30 20:44:00 UTC - ravi satya durga prasad Yenugula: 9527 is open I am
able to access it via localhost
----
2020-01-30 20:47:13 UTC - Ruslan Sheremet: In this case should be
REDIRECT_HOST=<http://localhost>
----
2020-01-30 21:00:12 UTC - David Kjerrumgaard: @ravi satya durga prasad Yenugula
The image is configured to use TLS for all connections, so you will need to use
https:// and port 8443 for the Pulsar REST interface, and pulsar+ssl:// and
port 6651 for producers and consumers
----
2020-01-30 21:01:27 UTC - ravi satya durga prasad Yenugula: where should I do
it?
----
2020-01-30 21:02:53 UTC - David Kjerrumgaard: I build and tag the image as
pia/pulsar-standalone-secure.
----
2020-01-30 21:03:01 UTC - David Kjerrumgaard: Then run it with this command
`docker run -id --name pulsar -p:6651:6651 -p 8443:8443 -p 80:80
--volume=${HOME}/exchange:/pulsar/manning/dropbox -t
pia/pulsar-standalone-secure:latest`
----
2020-01-30 21:04:29 UTC - David Kjerrumgaard: to get the TLS cert you need to
`docker exec -it pulsar bash` and run the following command to "publish" the
certs and JWT tokens to your local volume;
`/pulsar/manning/securtiy/publish-credentials.sh`
----
2020-01-30 21:05:16 UTC - ravi satya durga prasad Yenugula: Ok I will try and
let you know thanks. @Roman Popenov said to use a Manger Image which has UI is
it same or some thing different
----
2020-01-30 21:05:44 UTC - ravi satya durga prasad Yenugula: from the image
which you are saying
----
2020-01-30 21:06:31 UTC - David Kjerrumgaard: The Pulsar manager is a newer
tool, that is NOT included in the Docker image I use.
----
2020-01-30 21:19:26 UTC - David Lanouette: @David Lanouette has joined the
channel
----
2020-01-30 21:23:47 UTC - ravi satya durga prasad Yenugula: OMG there are so
many ports what do port 80 do?
----
2020-01-30 21:23:59 UTC - ravi satya durga prasad Yenugula: @David Kjerrumgaard
----
2020-01-30 21:24:42 UTC - David Kjerrumgaard: That used to be for the Pulsar
Dashboard, but that has been removed in 2.5.0. So you can ignore that for now
----
2020-01-30 21:24:56 UTC - ravi satya durga prasad Yenugula: Ok
----
2020-01-30 21:25:12 UTC - David Kjerrumgaard: The dashboard has been replaced
with the manager that @Roman Popenov mentioned
----
2020-01-30 21:25:24 UTC - ravi satya durga prasad Yenugula: Ok
----
2020-01-30 21:25:51 UTC - ravi satya durga prasad Yenugula: I am running it but
I see a issue
----
2020-01-30 21:25:55 UTC - David Kjerrumgaard: I am working on a docker-compose
now that will support both the secured pulsar image, and the new dashboard
component.
----
2020-01-30 21:25:58 UTC - ravi satya durga prasad Yenugula: not sure what I
need to do
----
2020-01-30 21:25:59 UTC - David Kjerrumgaard: what is the issue?
----
2020-01-30 21:26:05 UTC - ravi satya durga prasad Yenugula: I will add you to
the thread
----
2020-01-30 21:26:08 UTC - David Kjerrumgaard: ok
----
2020-01-30 21:26:29 UTC - ravi satya durga prasad Yenugula: @David Kjerrumgaard
please check my above screenshot
----
2020-01-30 21:30:49 UTC - David Kjerrumgaard: You need to change the
REDIRECT_HOST in the docker command you used to launch the manager.
----
2020-01-30 21:31:02 UTC - David Kjerrumgaard: to use localhost as suggested.
----
2020-01-30 21:32:14 UTC - ravi satya durga prasad Yenugula: I ran this
```docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e
REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e
URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e
PASSWORD=pulsar -e LOG_LEVEL=DEBUG apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
I am using that param
----
2020-01-30 21:32:57 UTC - Roman Popenov: ```REDIRECT_HOST=<http://localhost>```
or 127.0.0.1
----
2020-01-30 21:33:06 UTC - ravi satya durga prasad Yenugula: ok
----
2020-01-30 21:33:47 UTC - ravi satya durga prasad Yenugula: ok
----
2020-01-30 21:39:02 UTC - ravi satya durga prasad Yenugula: Server is up but
getting same issue some http error
----
2020-01-30 21:39:13 UTC - Joshua Dunham: Hi Everyone, How can I configure
Pulsar to utilize a 'remote' BookKeeper cluster?
+1 : Ryan
----
2020-01-30 21:39:17 UTC - ravi satya durga prasad Yenugula:
----
2020-01-30 21:39:45 UTC - ravi satya durga prasad Yenugula: ```docker run -it
-p 9527:9527 -e REDIRECT_HOST=<http://localhost> -e REDIRECT_PORT=9527 -e
DRIVER_CLASS_NAME=org.postgresql.Driver -e
URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e
PASSWORD=pulsar -e LOG_LEVEL=DEBUG apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
changed to localhost every 127...1
still I am getting that http error
----
2020-01-30 21:52:32 UTC - David Kjerrumgaard: The issue appears to be related
to the fact that you have specified a Postgres DB that is supposed to be
running on localhost but isn't
----
2020-01-30 21:53:00 UTC - ravi satya durga prasad Yenugula: so what should I do
now?
----
2020-01-30 21:58:55 UTC - Roman Popenov:
<https://github.com/apache/pulsar-manager> - have you tried following the steps
in the readme?
----
2020-01-30 22:57:34 UTC - rwaweber: @rwaweber has joined the channel
----
2020-01-30 23:16:15 UTC - David Kjerrumgaard: I haven't gotten this to work
yet. Now I am going to have to play with this a bit
----
2020-01-30 23:17:37 UTC - ravi satya durga prasad Yenugula: Please also let me
know your findings
----
2020-01-30 23:23:51 UTC - ravi satya durga prasad Yenugula: I tried the
environment endpoint as per the gif but I am getting response 404
----
2020-01-30 23:24:57 UTC - ravi satya durga prasad Yenugula: More over the
server is keep on throwing the error
```
<!doctype html><html lang="en"><head><title>HTTP Status 500 –
Internal Server Error</title><style type="text/css">h1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
h2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
h3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}
a {color:black;} a.name {color:black;} .line
{height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
Status 500 – Internal Server Error</h1></body></html>```
for the most of the clicks :disappointed:
----
2020-01-30 23:26:15 UTC - ravi satya durga prasad Yenugula: The setup is not
friendly :disappointed:
----
2020-01-30 23:26:47 UTC - Eugen: Yes, the compacted log will be written to
another, internal topic. The consumer can decide which one to consume from by
configuring `ConsumerBuilder.readCompacted(bool)`
----
2020-01-30 23:27:16 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula Do
you run pulsar itself?
----
2020-01-30 23:27:33 UTC - ravi satya durga prasad Yenugula: No
----
2020-01-30 23:34:00 UTC - David Kjerrumgaard: So I have it working now using
Docker compose. Here is the relevant file
----
2020-01-30 23:34:07 UTC - David Kjerrumgaard:
----
2020-01-30 23:35:04 UTC - David Kjerrumgaard: Save the above as
`docker-compose.yml` and just run `docker-compose up` from the same directory
where you save the file.
----
2020-01-30 23:35:11 UTC - ravi satya durga prasad Yenugula: youare using
standalone what is the difference between standalone and the master?
----
2020-01-30 23:35:53 UTC - David Kjerrumgaard: you should be able to go to
<http://localhost:9527> and manage your Pulsar cluster
----
2020-01-30 23:36:44 UTC - David Kjerrumgaard: The standalone image I am using
isn't secured like mine docker image is. So the manager can interact with it.
----
2020-01-30 23:37:14 UTC - David Kjerrumgaard: ^^ This is a limitation with the
manager, AFAIK
----
2020-01-30 23:37:36 UTC - ravi satya durga prasad Yenugula: So I have to run
both the manger and the standalone
----
2020-01-30 23:37:42 UTC - David Kjerrumgaard: yes
----
2020-01-30 23:39:39 UTC - ravi satya durga prasad Yenugula: ok what about the
config that between the master and the standalone box? is it preconfigured in
your image
----
2020-01-30 23:46:04 UTC - David Kjerrumgaard: yes
----
2020-01-30 23:46:36 UTC - David Kjerrumgaard: You will have to Add the
environment on the <http://localhost:9527/#/environments> link
----
2020-01-30 23:46:57 UTC - David Kjerrumgaard: just enter a name and give a URL
of "<http://pulsar:8080>"
----
2020-01-30 23:48:36 UTC - Alexandre DUVAL: Okay, cool!
----
2020-01-30 23:48:38 UTC - Alexandre DUVAL: Thanks
----
2020-01-30 23:54:38 UTC - Sijie Guo: what do you mean “remote”? an existing
bookkeeper cluster?
----
2020-01-30 23:56:00 UTC - Roman Popenov: @David Kjerrumgaard doing god’s work
:smile:
rolling_on_the_floor_laughing : David Kjerrumgaard
----
2020-01-31 00:15:02 UTC - Eugen: Anyone gotten this, trying to compact a topic?
Google hasn't;
```apache-pulsar-2.5.0$ bin/pulsar-admin topics compaction-status
<persistent://public/default/my-topic>
Error in compaction
null
Reason: Error compacting: java.util.concurrent.TimeoutException: Timeout```
----
2020-01-31 00:17:11 UTC - David Kjerrumgaard: @Eugen Are there any errors in
the broker log for that time period?
----
2020-01-31 00:29:04 UTC - Eugen: I just triggered compaction again
```$ bin/pulsar-admin topics compact <persistent://public/default/my-topic>```
And there were no ERR logs, only one perhaps unrelated WARN. Here are excerpts
starting from the first log of the compaction, including the WARN log and all
mentions of timeouts:
----
2020-01-31 00:29:04 UTC - Eugen: ```09:23:21.846 [pulsar-30-4] INFO
org.apache.pulsar.broker.namespace.OwnershipCache - Trying to acquire ownership
of public/default/0x40000000_0x80000000
09:23:21.860 [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO
org.apache.pulsar.broker.namespace.OwnershipCache - Successfully acquired
ownership of /namespace/public/default/0x40000000_0x80000000
09:23:21.860 [pulsar-30-11] INFO org.apache.pulsar.broker.PulsarService -
Loading all topics on bundle: public/default/0x40000000_0x80000000
09:23:21.862 [pulsar-ordered-OrderedExecutor-5-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger
public/default/persistent/my-topic
09:23:21.864 [bookkeeper-ml-workers-OrderedExecutor-2-0] WARN
org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [127.0.0.1:3181]
is not adhering to Placement Policy. quarantinedBookies: []
09:23:21.885 [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO
org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [127.0.0.1:3181] for
ledger: 1532
09:23:21.885 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[public/default/persistent/my-topic] Created ledger 1532
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor:
test-my-topic-2
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor:
test-1580373893803
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor:
test-1580374741644
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor:
test-1580374704326
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor:
__compaction
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor:
test-1580374820773
...
09:23:21.896 [bookkeeper-ml-workers-OrderedExecutor-2-0] WARN
org.apache.pulsar.broker.service.AbstractTopic -
[<persistent://public/default/my-topic>] Error getting policies
java.util.concurrent.CompletableFuture cannot be cast to
org.apache.pulsar.common.policies.data.Policies and publish throttling will be
disabled
09:23:21.896 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling
for <persistent://public/default/my-topic>
09:23:21.897 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.pulsar.broker.service.persistent.PersistentTopic -
[<persistent://public/default/my-topic>] There are no replicated subscriptions
on the topic
09:23:21.897 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.pulsar.broker.service.BrokerService - Created topic
<persistent://public/default/my-topic> - dedup is disabled
09:23:21.897 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO
org.apache.pulsar.broker.PulsarService - Loaded 1 topics on
public/default/0x40000000_0x80000000 -- time taken: 0.036 seconds
09:23:21.908 [pulsar-web-57-7] INFO
org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar
consumer status recorder with config: {
"topicNames" : [ "<persistent://public/default/my-topic>" ],
"topicsPattern" : null,
"subscriptionName" : "__compaction",
"subscriptionType" : "Exclusive",
"receiverQueueSize" : 1000,
"acknowledgementsGroupTimeMicros" : 100000,
"negativeAckRedeliveryDelayMicros" : 60000000,
"maxTotalReceiverQueueSizeAcrossPartitions" : 50000,
"consumerName" : null,
"ackTimeoutMillis" : 0,
"tickDurationMillis" : 1000,
"priorityLevel" : 0,
"cryptoFailureAction" : "FAIL",
"properties" : { },
"readCompacted" : true,
"subscriptionInitialPosition" : "Latest",
"patternAutoDiscoveryPeriod" : 1,
"regexSubscriptionMode" : "PersistentOnly",
"deadLetterPolicy" : null,
"autoUpdatePartitions" : true,
"replicateSubscriptionState" : false,
"resetIncludeHead" : false,
"keySharedPolicy" : null
}
...
09:23:22.006 [BookKeeperClientWorker-OrderedExecutor-5-0] INFO
org.apache.bookkeeper.mledger.impl.MetaStoreImplZookeeper -
[public/default/persistent/my-topic] [__compaction] Updating cursor info
ledgerId=1533 mark-delete=10:-1
09:23:22.025 [pulsar-timer-87-1] INFO
org.apache.pulsar.client.impl.ConnectionHandler -
[<persistent://public/default/my-topic>] [__compaction] Reconnecting after
timeout
...
09:23:22.362 [pulsar-client-io-85-1] INFO
org.apache.pulsar.compaction.TwoPhaseCompactor - Commencing phase one of
compaction for <persistent://public/default/my-topic>, reading to 10:3793:-1
09:23:23.594 [SessionTracker] INFO org.apache.zookeeper.server.ZooKeeperServer
- Expiring session 0x10537cac3d3000b, timeout of 30000ms exceeded```
----
2020-01-31 00:32:20 UTC - Eugen: btw is normal that pulsar only logs to stdout
when starting it like `bin/pulsar standalone`? It does create a
`logs/pulsar-standalone.log` file, but it does not write to it.
----
2020-01-31 00:34:24 UTC - David Kjerrumgaard: ^^^ Yes that is the expected
logging behavior. You can override it with the log4j settings
----
2020-01-31 00:34:50 UTC - Eugen: Ok so it's not an issue - never mind!
----
2020-01-31 00:34:50 UTC - Sijie Guo: bin/pulsar script starts component in
foreground. so it logs to stdout. because that will be the way how pulsar
component be running in a docker container.
the file create is probably related so log4j logger related to function worker.
+1 : Eugen
----
2020-01-31 00:36:10 UTC - David Kjerrumgaard: Nope, just a minor annoyance.
The compact error appears to be related to the ZK connection timing out
----
2020-01-31 00:37:49 UTC - Eugen: The compaction timeout occurred only a few
seconds after triggering it though, so the 30sec ZK timeout is at least not
synchronous to the compaction call
----
2020-01-31 00:39:35 UTC - Alexandre DUVAL: Got exactly the same issue by our
side
----
2020-01-31 00:40:07 UTC - David Kjerrumgaard: This looks like an issue. Can one
of you please file an issue on GitHub for this?
----
2020-01-31 00:40:24 UTC - Eugen: will do
+1 : David Kjerrumgaard, Alexandre DUVAL, Roman Popenov
----
2020-01-31 01:01:03 UTC - Eugen: I created this issue:
<https://github.com/apache/pulsar/issues/6173>
----
2020-01-31 02:04:21 UTC - Eugen: Has anyone tried to use Pulsar to build a
system that when a client starts consuming from a topic, it will want to get a
snapshot containing the latest value for every key, and then all real-time
messages coming in thereafter. Log compaction will only get you halfway there,
because it is asynchronous, so clients will receive duplicate keys for the
not-yet-compacted part.
----
2020-01-31 02:11:13 UTC - Eugen: Here's how it could perhaps be implemented:
1. have a Pulsar Function that consumes from a (not compacted) topic and
continuously updates the latest snapshot in a BookKeeper table, affixing the
SequenceId to that table
2. consumers will consume from the topic using InitialPosition `Latest`, taking
note of the first SequenceId they receive
3. consumers will get a snapshot from the BookKeeper table with a SequenceId
>= that of the first message received via the consumer and merge the
snapshot with the messages consumed in real-time
I have not worked with the BookKeeper table service, so I don't know if this is
feasible or not. Any ideas?
----
2020-01-31 02:19:13 UTC - Eugen: There are at least 2 ways to do the merging,
depending of whether you need to have strict ordering of messages from
different keys or not (I would expect order within a single key to always be
honored)
----
2020-01-31 03:21:47 UTC - Sijie Guo: The abstraction of this problem is for
Pulsar to provide a materialized table view of an “event stream”.
The topic compaction approach is half-through, since it only creates a
point-in-time snapshot. To do one step future, we can just implement a
“compact/merge-at-read” approach. i.e when processing the compacted view and
the uncompacted events, merge them at read time.
The approach you are proposing here will be an approach that is more like a
“merge-at-write” approach. as you are materializing the view at the write time
to the bookkeeper table.
Both approaches have pros and cons. It depends on what kind of queries you want
to support on this “materialized” view.
----
2020-01-31 04:00:52 UTC - Eugen: It seems I have not expressed this clear
enough. My approach was definitely meant as a merge-at-read approach, and what
I called "snapshot", is what you call materialized table view.
----
2020-01-31 05:45:08 UTC - Sijie Guo: ah okay. then we are on the same page.
----
2020-01-31 05:50:01 UTC - Eugen: by "materialized table view of an event
stream", are you referring to the BookKeeper table service?
----
2020-01-31 06:12:57 UTC - Sijie Guo: @Eugen I was referring to a general
concept. I didn’t refer to a specific implementation. the main question would
be what are the expected operations over this materialized table. is it scan or
point lookup? because that can result in different design choices.
----
2020-01-31 06:14:05 UTC - Eugen: it would be scan, as I want to get all
key/values
----
2020-01-31 06:15:08 UTC - Eugen: however, it does not technically need to be a
real "snapshot" (at one point in time), as long as I have the SequenceId for
every key/value pair that I'm scanning
----
2020-01-31 06:19:17 UTC - Ravi Shah: Hi Team, Can i install pulsar c++ client
on Alpine?
----
2020-01-31 06:26:52 UTC - Sijie Guo: If it is a scan, you don’t need to
materialize to bookkeeper table. since you have a compacted topic (up to given
messageId), you can provide a consistent view by reading the compacted topic
and merging it with the uncompacted data.
----
2020-01-31 06:33:03 UTC - Eugen: Hm, I'm still debating whether or not to use
compaction for this. Will need to play with it to get a feel for it.
----
2020-01-31 06:39:47 UTC - Sijie Guo: my feeling is if your operation is to get
all the kvs at given point of time. compaction is probably an efficient way.
but yet. streaming the events to build a table can be an alternative way
----
2020-01-31 07:01:29 UTC - Eugen: Thanks for your input! Will check it out, and
if we will end up using Pulsar, I will share my experience here in a couple of
weeks to a few months
+1 : Sijie Guo
----
2020-01-31 07:06:26 UTC - Sijie Guo: feel free to ping me if you need any help
heart : Eugen
----
2020-01-31 07:12:28 UTC - Ali Ahmed: @Ravi Shah yes it can be done needs some
specific setup I don’t think it’s documented yet
----
2020-01-31 07:20:09 UTC - Ravi Shah: @Ali Ahmed Do you have any link to guide
which i can refer or any sort of documentation?
----
2020-01-31 07:20:56 UTC - Ali Ahmed: I can find some time over the weekend to
compile it, don’t have anything to share right now
----
2020-01-31 07:27:06 UTC - Ravi Shah: That will be helpful if you can compile it
provide. By any chance you know when it will be documented?
----
2020-01-31 07:29:16 UTC - Ali Ahmed: will have to find some time hopefully have
something to share next week
----
