2019-02-07 09:23:30 UTC - Olaf Thormählen: @Olaf Thormählen has joined the channel ---- 2019-02-07 15:47:59 UTC - Emma Pollum: agmentReplicator - Replicating fragment Fragment(LedgerID: 4231, FirstEntryID: 0[0], LastKnownEntryID: 21[21], Host: [10.128.2.3:3181, 10.128.1.168:3181], Closed: true) in 1 sub fragments. Feb 07 15:44:33 p-pulsar-central1-3 bookkeeper[8341]: 15:44:33.285 [bookkeeper-io-16-6] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0x6b33b837, L:/10.128.4.137:48724]/10.128.1.168:3181, current state CONNECTING : ---- 2019-02-07 19:35:53 UTC - Ali Ahmed: the pulsar cpp lib has ported the buckaroo c++ package manger , this should allow it to run across platforms <https://github.com/njlr/apache-pulsar-demo> ---- 2019-02-07 20:43:05 UTC - Ambud Sharma: The PulsarMessageRouter.choosePartition() method doesn't seem to be getting called ```producer = pulsarClient.newProducer() .compressionType( CompressionType.valueOf(producerConfig.getCompressionType().toUpperCase())) .messageRoutingMode(MessageRoutingMode.CustomPartition) .messageRouter(new PulsarMessageRouter( (Class<Partitioner>) Class.forName(producerConfig.getPartitionerClass()))) .sendTimeout(producerConfig.getWriteTimeoutInSeconds(), TimeUnit.SECONDS).topic(topic)``` ---- 2019-02-07 20:59:48 UTC - Ambud Sharma: the issue seems to be using the correct URL for topic to have the partitioner be called, I think we should add something to block automatic topic production; ---- 2019-02-07 21:00:00 UTC - Ambud Sharma: is there a way to disable automatic topic creation? ---- 2019-02-07 21:31:23 UTC - Ambud Sharma: how do we achieve partition-level message-ordering when using a partitioned topic for consumers? as per documentation shared subscription doesn't seem to honor message ordering ---- 2019-02-07 22:11:46 UTC - Ivan Kelly: @Ambud Sharma there's an effort in place to disable automatic creation right now ---- 2019-02-07 22:17:15 UTC - Ambud Sharma: thanks; <https://github.com/apache/pulsar/pull/3450> just to confirm ---- 2019-02-07 22:55:16 UTC - Matteo Merli: @Ambud Sharma Using failover subscription will get 1 active consumer on each partition ---- 2019-02-07 22:55:42 UTC - Matteo Merli: similar to Kafka consumer group ---- 2019-02-07 22:58:46 UTC - Ambud Sharma: thanks @Matteo Merli ---- 2019-02-07 23:25:27 UTC - Emma Pollum: Even while idle (no messages processing through the system, but there are messages in topics, just no consumers or producers currently) my bookies are taking up a whopping 19G of memory. Does this sound right? ---- 2019-02-08 00:06:46 UTC - Sijie Guo: yes that’s the issue ---- 2019-02-08 00:08:13 UTC - Sijie Guo: @Emma Pollum can you share how do you set PULSAR_MEM in your config? ---- 2019-02-08 00:25:48 UTC - Matteo Merli: Also the `bookkeeper.conf`, any setting different from the defaults ---- 2019-02-08 01:39:33 UTC - David Kjerrumgaard: This sequence of messages indicates that the fragment was replicated to hosts 10.128.2.3 & 10.128.1.168 (which I am assuming are the two bookies you removed). The next message indicates that BK was unable to connect to host 10.128.4.137. Is that the IP of one of the remaining bookies in your cluster? ---- 2019-02-08 07:28:03 UTC - Niketh Sabbineni: @Niketh Sabbineni has joined the channel ---- 2019-02-08 07:29:57 UTC - Niketh Sabbineni: Is the message Id enough for a successful acknowledgement? ``` Message fromPular = getFomPulsar(). consumer.ackowledge(new Message(fromPulsar.getMessageId())); ``` ---- 2019-02-08 07:31:35 UTC - Niketh Sabbineni: Using pulsar shared subscription, i need to buffer the messages till my process completes, which could be an hr. I don't want to buffer all the records in memory, seeing if I can get rid of the payload and just store the messageIds. When i am ready, i construct the required message object, using the stored messageIds; ---- 2019-02-08 07:35:38 UTC - Sijie Guo: @Niketh Sabbineni - you can ack using message id `void acknowledge(MessageId messageId) throws PulsarClientException;` ---- 2019-02-08 08:01:58 UTC - dba: Are window functions supported in Python? If yes, could you link to an example? ----
