2018-11-09 10:55:04 UTC - tuteng: ./pulsar/bin/pulsar-admin source localrun
--archive /data/pulsar/pulsar/pulsar-io/canal/target/pulsar-io-canal.jar
--classname org.apache.pulsar.canal.CanalSource --tenant public --namespace
default --name canal --source-config-file
/data/pulsar/canal-source-config-file.yml
Exception in thread "main" com.beust.jcommander.ParameterException: Unknown
option: --this$0
at com.beust.jcommander.JCommander.parseValues(JCommander.java:742)
at com.beust.jcommander.JCommander.parse(JCommander.java:282)
at com.beust.jcommander.JCommander.parse(JCommander.java:265)
at
org.apache.pulsar.functions.runtime.LocalRunner.main(LocalRunner.java:87)
----
2018-11-09 10:55:42 UTC - tuteng: What is the problem?
----
2018-11-09 10:56:28 UTC - tuteng: standalone mode
----
2018-11-09 10:58:01 UTC - tuteng: myself source:
<https://github.com/AmateurEvents/pulsar/tree/feature/pulsar-io-support-canal/pulsar-io/canal/src/main/java/org/apache/pulsar/canal>
----
2018-11-09 10:58:32 UTC - Ivan Kelly: where is canal-source-config-file.yml
----
2018-11-09 10:59:53 UTC - tuteng: local file: configs:
zkServers: "10.4.239.22:2181"
batchSize: 5120
destination: "example"
username: ""
password: ""
----
2018-11-09 11:15:08 UTC - Ivan Kelly: anything in ./logs?
----
2018-11-09 11:16:57 UTC - Ivan Kelly: can you try with PULSAR_LOG_LEVEL=DEBUG
in the environment
----
2018-11-09 12:40:25 UTC - tuteng: logs are normal. No information received for
pulsar broker
----
2018-11-09 12:50:21 UTC - Ivan Kelly: what version is this?
----
2018-11-09 13:07:20 UTC - tuteng: I'm developing on the master branch, and I
don't know which version. Sorry, this is history commit
<https://github.com/AmateurEvents/pulsar/commits/feature/pulsar-io-support-canal>
----
2018-11-09 13:08:41 UTC - Ivan Kelly: Ok, are you building the whole pulsar
repo each time then?
----
2018-11-09 13:11:08 UTC - tuteng: yes, use the command mvn clean install
-DskipTests
----
2018-11-09 13:12:01 UTC - tuteng: Today is the first compilation and
installation.
----
2018-11-09 13:12:31 UTC - Ivan Kelly: Ok, can you put a log line in
LocalRunner.main to print out the full commandline?
----
2018-11-09 13:18:47 UTC - tuteng: OK, I'll try. Thank you very much.
----
2018-11-09 13:19:49 UTC - Ivan Kelly: it's possible a change to functions may
have broken localrun for I/O so let me know if you find anything
----
2018-11-09 13:57:53 UTC - Grégory Guichard: @Grégory Guichard has joined the
channel
----
2018-11-09 14:35:40 UTC - jia zhai: Oh, IC @Ivan Kelly @tuteng
<https://github.com/apache/pulsar/pull/2892>
There was a change to these command in master
----
2018-11-09 16:42:41 UTC - Matteo Merli: @tuan nguyen anh it’s strange. On my
laptop with the default config the throughput is ~300K msg/s (300 MBytes/s) :
```
08:40:36.080 [main] INFO org.apache.pulsar.testclient.PerformanceProducer -
Throughput produced: 297539.6 msg/s --- 2324.5 Mbit/s --- Latency: mean:
2.723 ms - med: 2.100 - 95pct: 4.679 - 99pct: 7.302 - 99.9pct: 17.603 -
99.99pct: 719.831 - Max: 721.191
08:40:46.168 [main] INFO org.apache.pulsar.testclient.PerformanceProducer -
Throughput produced: 313669.3 msg/s --- 2450.5 Mbit/s --- Latency: mean:
2.969 ms - med: 1.684 - 95pct: 4.267 - 99pct: 8.664 - 99.9pct: 43.156 -
99.99pct: 1462.783 - Max: 1463.527
08:40:56.020 [pulsar-perf-producer-exec-1-1] INFO
org.apache.pulsar.testclient.PerformanceProducer - ------------------- DONE
-----------------------
08:40:56.051 [pulsar-perf-producer-exec-1-1] INFO
org.apache.pulsar.testclient.PerformanceProducer - Aggregated latency stats ---
Latency: mean: 3.010 ms - med: 1.808 - 95pct: 4.784 - 99pct: 9.524 -
99.9pct: 106.352 - 99.99pct: 1461.311 - 99.999pct: 1463.439 - Max: 1463.527
```
----
2018-11-09 17:02:39 UTC - tuan nguyen anh: amazing, i don't understand why this
so my solution isiun it macbook
----
2018-11-09 17:34:12 UTC - Matteo Merli: @tuan nguyen anh Can you open an issue
on github and recap all your steps?
----
2018-11-09 19:13:36 UTC - Emma Pollum: What is the best way to give the JVM
more more memory for the broker?
----
2018-11-09 19:15:08 UTC - Matteo Merli: edit `conf/pulsar_env.sh` :
`PULSAR_MEM=${PULSAR_MEM:-"-Xms8g -Xmx8g -XX:MaxDirectMemorySize=16g"}`
----
2018-11-09 19:15:41 UTC - Emma Pollum: Thank you!
----
2018-11-09 21:44:05 UTC - Beast in Black: @Matteo Merli as a followup to all
those annoying questions I asked yesterday, I have a couple more questions
which popped up when I was making the change to have my consumer type be
shared. These questions are due to what I understand is the round-robin nature
of the `shared` consumer type and the documentation for the same available at
`<https://pulsar.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/#Exclusive-u84f1>`
*To recap:* In my use case, each app instance (C++ app using the pulsar CPP
client) will subscribe to a global topic (persistent as well as non-persistent)
using a unique consumer subscription name i.e. each app instance will use a
unique subscription name which is different from all the rest of the app
instances. This unique name is persisted on each app instance so when the app
comes back after a crash it will continue to use the same subscription name as
before.
Thus, on each app instance there will be exactly one consumer and one
subscriber for the topic, and this consumer's type will be set to `shared` to
allow the app to restart after a crash and to re-subscribe to the topic using
the same consumer name it used as before without experiencing pulsar client
errors related to trying to resubscribe to an existing (i.e unreaped) broker
consumer connection when using the same subscription name.
*Here come the questions:*
1. If new messages were published to that topic while the app instance was
offline due to the crash/restart, when it connects back using the same
subscription name to the same shared consumer (to which no other subscriptions
will ever connect), will it receive/consume the messages which it missed while
it was down?
2. After coming back up from the crash, will it receive/consume all new
messages published after it came back up? Essentially, the question asks
whether the round-robin nature of shared consumers will cause message receipt
issues when using the same subscription name for the same shared consumer after
an app crash/restart.
3. On each app instance, since there is only a single consumer/subscriber for
the given topic, can message ordering still be guaranteed despite the
round-robin nature of a shared consumer?
----
2018-11-09 22:47:50 UTC - Sanjeev Kulkarni: @tuteng The fix to your problem is
at <https://github.com/apache/pulsar/pull/2970>
----
2018-11-10 00:34:31 UTC - Libin: @Libin has joined the channel
----
