2020-01-28 09:16:27 UTC - juraj: i have single partitioned topic.
but i have create a simple test case, and it seems that there is a bug ---
cumulative acks get ignored on messages older than even 15 seconds
----
2020-01-28 09:17:17 UTC - juraj: single topic, failover subscription, single
producer, single consumer
----
2020-01-28 10:05:01 UTC - juraj: <https://github.com/apache/pulsar/issues/6157>
----
2020-01-28 12:09:21 UTC - Alexandre DUVAL: Hi, function `runtimeFlags`
configuration enables to pass env vars to function exe. But for creds there is
a way to pass something like `runtimeFlags: -Dsecret=${SECRET}` with
`${SECRET}` defined at system env var level?
----
2020-01-28 16:20:15 UTC - Guilherme Perinazzo: Is there anyway to disable stat
collection on the go client? Setting it to 0 doesn't work
----
2020-01-28 16:30:13 UTC - Alexandre DUVAL: @Sijie Guo
----
2020-01-28 16:42:58 UTC - Joshua Dunham: Hey Everyone, I'm deploying on via
something like docker-compose.yml but can't find the config options for pulsar
defining the location of the bookies.
----
2020-01-28 16:43:08 UTC - Joshua Dunham: pointers?
----
2020-01-28 18:04:28 UTC - Joshua van Eck: @Joshua van Eck has joined the channel
----
2020-01-28 19:37:22 UTC - Roman Popenov: Does anyone have documentation on to
how to configure the stream for prometheus jobs?
----
2020-01-28 19:37:47 UTC - Roman Popenov: Or knows how to set-up prometheus
properly for monitoring so the dashboard picks up the metrics?
----
2020-01-28 20:07:24 UTC - Mathieu Druart: Hi ! I try to use Pulsar (2.5.0) with
Spark (2.4.4) and the <https://github.com/streamnative/pulsar-spark> connector.
Pulsar is deployed on k8s, and Spark is launched locally.
----
2020-01-28 20:07:39 UTC - Mathieu Druart: I tried this code :
----
2020-01-28 20:07:45 UTC - Mathieu Druart: ``` String serviceUrl =
"<pulsar://172.16.101.114:30002>";
String adminUrl = "<http://172.16.101.114:30001>";
SparkSession spark =
SparkSession.builder().appName("PulsarTest").master("local[2]").getOrCreate();
spark.sparkContext().setLogLevel("WARN");
Dataset<Row> dataset = spark
.readStream()
.format("pulsar")
.option("service.url", serviceUrl)
.option("admin.url", adminUrl)
.option("topic", "aom-emp")
.load();
StreamingQuery query = dataset
.withWatermark("__eventTime", "10 minutes")
.groupBy(window(dataset.col("__eventTime"), "1 minute"),
col("objet"), col("action"))
.sum("nb")
.select(
concat(col("objet"), lit("-"),
col("action")).as("__topic"),
col("window").cast(DataTypes.StringType).as("__key"),
col("sum(nb)").as("value"))
.writeStream()
.outputMode("update")
.format("pulsar")
.option("service.url", serviceUrl)
.option("admin.url", adminUrl)
.option("checkpointLocation", "C:\\tmp\\checkpoint")
.start();
query.awaitTermination();```
----
2020-01-28 20:08:40 UTC - Mathieu Druart: but I have randomly this error :
`org.apache.pulsar.client.api.PulsarClientException: Error when resetting
subscription: 142926:2` , any Idea ?
----
2020-01-28 22:31:30 UTC - Bobby: @Bobby has joined the channel
----
2020-01-28 22:32:20 UTC - Bobby: Hi all, we are trying to run bookkeeper in
kube, and running into issues with configuration. The bookkeeper nodes keep
starting up with default values, and they're not taking our changes in the
configmap that was created. Any ideas?
----
2020-01-28 22:32:53 UTC - Bobby:
----
2020-01-28 22:34:48 UTC - Bobby:
----
2020-01-28 22:36:03 UTC - Roman Popenov: Which configurations exactly? For
direct mem related?
----
2020-01-28 22:36:38 UTC - Bobby: right now, the bookie won't start because it
can't connect to zookeeper
----
2020-01-28 22:36:48 UTC - chris: try removing the PULSAR_PREFIX_
----
2020-01-28 22:36:48 UTC - Bobby: but we can't specify in the configmap where
they are with the bookies picking it up
----
2020-01-28 22:36:57 UTC - Bobby: ok one sec
----
2020-01-28 22:37:24 UTC - chris: that prefix should only be used if the value
is not in the conf or is commented out by default
----
2020-01-28 22:40:06 UTC - Bobby: oh cool, yeah it's seeing the zookeepers now.
----
2020-01-28 22:41:06 UTC - Bobby: i'm getting an invalid cookie error now, i'm
not sure what that means
----
2020-01-28 22:42:52 UTC - Bobby:
----
2020-01-28 22:45:11 UTC - Roman Popenov: I thought the config was
```zkServers:```
----
2020-01-28 22:45:40 UTC - Roman Popenov: But that didn’t work for me
:thinking_face:
----
2020-01-28 22:46:25 UTC - ravi satya durga prasad Yenugula: The step from the
document is not working
----
2020-01-28 23:34:10 UTC - Pradeesh: does anyone know what the invalid cookie
exception in bookie is about ?
----
2020-01-28 23:34:14 UTC - Pradeesh: ```22:41:37.751 [main] ERROR
org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4
bookieHost: "192.168.65.247:3181"
journalDir: "data/bookkeeper/journal"
ledgerDirs: "1\tdata/bookkeeper/ledgers"
instanceId: "01847424-6122-4a5e-ae7d-1f835fe6121e"
] is not matching with [4
bookieHost: "bookkeeper-0.bookkeeper.pulsar.svc.cluster.local:3181"
journalDir: "data/bookkeeper/journal"
ledgerDirs: "1\tdata/bookkeeper/ledgers"
instanceId: "01847424-6122-4a5e-ae7d-1f835fe6121e"
]
at org.apache.bookkeeper.bookie.Cookie.verifyInternal(Cookie.java:136)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Cookie.verify(Cookie.java:147)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.Bookie.readAndVerifyCookieFromRegistrationManager(Bookie.java:348)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:420)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:250)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:688)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:105)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:41)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.doMain(Main.java:221)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.main(Main.java:203)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:313)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]```
----
2020-01-28 23:43:40 UTC - Sijie Guo: Currently it doesn’t take system env vars
yet.
----
2020-01-28 23:43:49 UTC - Sijie Guo: Can you create a github issue for that?
----
2020-01-28 23:44:12 UTC - Sijie Guo: which one are you using? c-go client or
native go client?
----
2020-01-28 23:44:58 UTC - Guilherme Perinazzo: c-go
----
2020-01-28 23:45:51 UTC - Guilherme Perinazzo: The code checks if it's
different than 0, but as far as i remember the c client expects 0 if you want
to disable it?
----
2020-01-28 23:46:25 UTC - Sijie Guo:
<https://github.com/streamnative/apache-pulsar-grafana-dashboard#prometheus>
There is an instructions for how to configure prometheus.
----
2020-01-28 23:47:22 UTC - Sijie Guo: the locations for bookie directories are
under directory `data/bookkeeper`. You just need to make sure your docker
compose file to mount the volumes correctly.
----
2020-01-28 23:48:31 UTC - Sijie Guo: Can you file an github issue to that repo?
@yijie can help you with the issue.
----
2020-01-28 23:51:20 UTC - Sijie Guo: because the bookie tried to start with ip
address before (PULSAR_PREFIX_useHostNameAsBookieID doesn’t take effect) which
left a cookie with ip address.
----
2020-01-28 23:51:51 UTC - Sijie Guo: when you try to start it with hostname,
bookie will fail to startup.
----
2020-01-28 23:52:20 UTC - Roman Popenov: I didn’t quire understand how to
easily set up the ipd/servicenames for the jobs:
```- targets:
# TODO: add the proxies to monitor
#
# - 'proxy1:8080'
# - 'proxy2:8080'
# - ...```
----
2020-01-28 23:52:46 UTC - Sijie Guo: try to delete the bookie stateful set and
the pvs created by the job
----
2020-01-28 23:53:12 UTC - Sijie Guo: and then re-create the statefulset
----
2020-01-28 23:56:21 UTC - Sijie Guo: If you are using windows, the instructions
are for linux containers in windows. Please check:
<https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-10-linux>
----
2020-01-28 23:59:13 UTC - Sijie Guo: @Pradeesh
The exception means that you started bookie before using IP address. Bookie
left a cookie with IP address on the disks. When you switched to useHostName,
it changed the bookie identifier, so bookie refuses to start.
Is this a production environment? If it is not a production environment,
1. delete the bookie statefulset.
2. delete the pvs used by the bookie statefulset.
3. recreate the bookie statefulset.
----
2020-01-28 23:59:38 UTC - Sijie Guo: That’s for on-premise deployment.
----
2020-01-28 23:59:47 UTC - Sijie Guo: Are you on kubernetes?
----
2020-01-28 23:59:57 UTC - Roman Popenov: On k8s
----
2020-01-29 00:01:22 UTC - Pradeesh: so all the pvc’s should be deleted
----
2020-01-29 00:01:27 UTC - Pradeesh: ```journal-disk-bookkeeper-0 Bound
pvc-b78d0cd5-3eee-11ea-a0c6-005056907a3f 20Gi RWO
rdei-sequential-io-storage 4d2h
journal-disk-bookkeeper-1 Bound pvc-c362d7b6-3eee-11ea-a0c6-005056907a3f
20Gi RWO rdei-sequential-io-storage 4d2h
journal-disk-bookkeeper-2 Bound pvc-d2f619a2-3eee-11ea-a0c6-005056907a3f
20Gi RWO rdei-sequential-io-storage 4d2h
ledgers-disk-bookkeeper-0 Bound pvc-b7916103-3eee-11ea-a0c6-005056907a3f
20Gi RWO rdei-sequential-io-storage 4d2h
ledgers-disk-bookkeeper-1 Bound pvc-c38abee7-3eee-11ea-a0c6-005056907a3f
20Gi RWO rdei-sequential-io-storage 4d2h
ledgers-disk-bookkeeper-2 Bound pvc-d2f9f249-3eee-11ea-a0c6-005056907a3f
20Gi RWO rdei-sequential-io-storage 4d2h```
----
2020-01-29 00:01:32 UTC - Pradeesh: cool will try that
----
2020-01-29 00:03:54 UTC - Sijie Guo: okay configuring a prometheus on
kubernetes is much easier. it just scrape metrics by pods with annotations.
If you are using the helm chart, it will have the promethues configured
already:
<https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/templates/prometheus-configmap.yaml>
----
2020-01-29 00:04:34 UTC - Roman Popenov: Alright, I will try
----
2020-01-29 00:06:04 UTC - Roman Popenov: I did add this to the chart:
``` scrape_interval: 10s
evaluation_interval: 10s
external_labels:
cluster: {{ template "pulsar.fullname" . }}```
I am not sure if it is necessary, but before adding that, it wasn’t working
properly
----
2020-01-29 00:06:25 UTC - Roman Popenov: Also, the topic dashboard is not
loading properly for some reason
----
2020-01-29 00:07:05 UTC - Roman Popenov: ```Templating init failed
Datasource named ${DS_default} was not found```
----
2020-01-29 00:08:32 UTC - Pradeesh: @Sijie Guo so I deleted the pvc’s which
deletes the pv’s and also deleted the statefulset and recreated it. But running
into the same issue still
----
2020-01-29 00:09:14 UTC - Pradeesh: ```journal-disk-bookkeeper-0 Bound
pvc-260d8449-422b-11ea-84d1-00505690ed3a 20Gi RWO
rdei-sequential-io-storage 2m53s
journal-disk-bookkeeper-1 Bound pvc-36327e30-422b-11ea-84d1-00505690ed3a
20Gi RWO rdei-sequential-io-storage 2m26s
journal-disk-bookkeeper-2 Bound pvc-496c2759-422b-11ea-84d1-00505690ed3a
20Gi RWO rdei-sequential-io-storage 113s
ledgers-disk-bookkeeper-0 Bound pvc-260fd779-422b-11ea-84d1-00505690ed3a
20Gi RWO rdei-sequential-io-storage 2m53s
ledgers-disk-bookkeeper-1 Bound pvc-3634a63a-422b-11ea-84d1-00505690ed3a
20Gi RWO rdei-sequential-io-storage 2m26s
ledgers-disk-bookkeeper-2 Bound pvc-4969b1fb-422b-11ea-84d1-00505690ed3a
20Gi RWO rdei-sequential-io-storage 113s```
----
2020-01-29 00:09:34 UTC - Sijie Guo: oh
----
2020-01-29 00:09:50 UTC - Pradeesh: ```5.275: Total time for which application
threads were stopped: 0.0000283 seconds, Stopping threads took: 0.0000108
seconds
5.275: Total time for which application threads were stopped: 0.0000702
seconds, Stopping threads took: 0.0000517 seconds
00:08:35.629 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build
bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4
bookieHost: "192.168.78.120:3181"
journalDir: "data/bookkeeper/journal"
ledgerDirs: "1\tdata/bookkeeper/ledgers"
instanceId: "01847424-6122-4a5e-ae7d-1f835fe6121e"
] is not matching with [4
bookieHost: "bookkeeper-0.bookkeeper.pulsar.svc.cluster.local:3181"
journalDir: "data/bookkeeper/journal"
ledgerDirs: "1\tdata/bookkeeper/ledgers"
instanceId: "01847424-6122-4a5e-ae7d-1f835fe6121e"
]
at org.apache.bookkeeper.bookie.Cookie.verifyInternal(Cookie.java:136)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Cookie.verify(Cookie.java:147)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.Bookie.readAndVerifyCookieFromRegistrationManager(Bookie.java:348)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:420)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:250)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:688)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:105)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:41)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.doMain(Main.java:221)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.main(Main.java:203)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:313)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
5.279: Total time for which application threads were stopped: 0.0000458
seconds, Stopping threads took: 0.0000112 seconds
5.279: Total time for which application threads were stopped: 0.0000231
seconds, Stopping threads took: 0.0000059 seconds
5.279: Total time for which application threads were stopped: 0.0000204
seconds, Stopping threads took: 0.0000051 seconds
5.279: Total time for which application threads were stopped: 0.0000198
seconds, Stopping threads took: 0.0000049 seconds
5.279: Total time for which application threads were stopped: 0.0000259
seconds, Stopping threads took: 0.0000066 seconds
5.279: Total time for which application threads were stopped: 0.0000199
seconds, Stopping threads took: 0.0000049 seconds
5.279: Total time for which application threads were stopped: 0.0000192
seconds, Stopping threads took: 0.0000050 seconds
Heap```
----
2020-01-29 00:11:15 UTC - Pradeesh: @Sijie Guo any other ideas?
----
2020-01-29 00:36:11 UTC - Sijie Guo: oh I forget one thing, you need to clean
up the zookeeper znodes.
----
2020-01-29 00:36:31 UTC - Pradeesh: yep I did that and got it to work past the
cookie ID exception
----
2020-01-29 00:36:37 UTC - Pradeesh: but now have a new issue on the bookies
----
2020-01-29 00:36:47 UTC - Pradeesh: ```00:32:41.472 [BookieJournal-3181] ERROR
org.apache.bookkeeper.common.component.ComponentStarter - Triggered
exceptionHandler of Component: bookie-server because of Exception in Thread:
Thread[BookieJournal-3181,5,main]
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216
byte(s) of direct memory (used: 2147483648, max: 2147483648)```
----
2020-01-29 00:36:57 UTC - Pradeesh: @Sijie Guo ^^
----
2020-01-29 00:41:07 UTC - Sijie Guo: @Pradeesh what is your bookie configmap?
----
2020-01-29 00:47:31 UTC - Sijie Guo: @Pradeesh if you configure `PULSAR_MEM`
for you bookie, you might consider adding `BOOKIE_MEM` as well. because
starting from 2.5.0, pulsar script starts taking some bookie related
environment settings when running `bookie` component. we will need to update
the helm script and documentation about this. at the same time, just add
`BOOKIE_MEM` into your bookie configmap should solve your problem.
----
2020-01-29 00:49:02 UTC - Sijie Guo: which topic dashboard? “dashboard” is a
over-used term.
----
2020-01-29 00:49:17 UTC - Sijie Guo: you mean grafana?
----
2020-01-29 00:49:27 UTC - Sijie Guo: or pulsar dashboard?
----
2020-01-29 00:49:34 UTC - Roman Popenov: grafana dashboards
----
2020-01-29 00:58:30 UTC - Sijie Guo: I see. for the topic dashboard, you need
to change the datasource in grafana to point to the cluster. because the image
doesn’t know your cluster name.
----
