[GitHub] jiazhai opened a new pull request #2583: Issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener

2018-09-14 Thread GitBox
jiazhai opened a new pull request #2583: Issue #2574: Timeout message not get 
redeliver in TopicsConsumer when use message listener 
URL: https://github.com/apache/incubator-pulsar/pull/2583
 
 
   ### Motivation
   
   fix issue #2574 .
   Timeout message not get redeliver in TopicsConsumer when use message 
listener.
   This is caused by message listener wrongly set in individual 
sub-ConsumerImpl.
   
   ### Modifications
   
   set message listener to null for individual sub-ConsumerImpl.
   Add a UT
   
   ### Result
   
   UT passed.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] jiazhai opened a new issue #2584: unacked message is not redelivered in setting ackTimeout

2018-09-14 Thread GitBox
jiazhai opened a new issue #2584: unacked message is not redelivered in setting 
ackTimeout
URL: https://github.com/apache/incubator-pulsar/issues/2584
 
 
    Expected behavior
   
   unacked message is not redelivered in setting ackTimeout
   
    Actual behavior
   unacked message is not redelivered in setting ackTimeout
   
    Steps to reproduce
   - create topic, producer and consumer, set consumer acktimeout
   - call consumer.receive(), and not ack the received message.
   - expected the message be redelivered after acktimeout; but actually not. 
   it is actually redelivered after 2*acktimeout
   
    System configuration
   2.1


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2576: Pulsar client consume command fails with java.lang.IllegalArgumentException: port out of range:-1

2018-09-14 Thread GitBox
sijie commented on issue #2576: Pulsar client consume command fails with 
java.lang.IllegalArgumentException: port out of range:-1
URL: 
https://github.com/apache/incubator-pulsar/issues/2576#issuecomment-421408473
 
 
   @dsambandam : what are the command you are using for initializing cluster 
metadata?
   
   especially what did you provide for `--web-service-url-tls` and 
`--broker-service-url-tls`?
   
   ```
   $ bin/pulsar initialize-cluster-metadata \
 --cluster pulsar-cluster-1 \
 --zookeeper zk1.us-west.example.com:2181 \
 --configuration-store zk1.us-west.example.com:2181 \
 --web-service-url http://pulsar.us-west.example.com:8080 \
 --web-service-url-tls https://pulsar.us-west.example.com:8443 \
 --broker-service-url pulsar://pulsar.us-west.example.com:6650 \
 --broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] aahmed-se commented on issue #2571: Change repository tag for test image

2018-09-14 Thread GitBox
aahmed-se commented on issue #2571: Change repository tag for test image
URL: https://github.com/apache/incubator-pulsar/pull/2571#issuecomment-421413403
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2500: [schema] add schemas for primtive types

2018-09-14 Thread GitBox
sijie commented on issue #2500: [schema] add schemas for primtive types
URL: https://github.com/apache/incubator-pulsar/pull/2500#issuecomment-421409154
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] jiazhai commented on issue #2584: unacked message is not redelivered in setting ackTimeout

2018-09-14 Thread GitBox
jiazhai commented on issue #2584: unacked message is not redelivered in setting 
ackTimeout
URL: 
https://github.com/apache/incubator-pulsar/issues/2584#issuecomment-421387939
 
 
   @codelipenghui meet this issue in their usage. 
   The main reason is in UnAckedMessageTracker.
   ```
   public void start(PulsarClientImpl client, ConsumerBase consumerBase, 
long ackTimeoutMillis) {
   this.stop();
   timeout = client.timer().newTimeout(new TimerTask() {
   @Override
   public void run(Timeout t) throws Exception {
   if (isAckTimeout()) {   < === first timeout, it is false, 
because oldOpenSet is empty.
   log.warn("[{}] {} messages have timed-out", 
consumerBase, oldOpenSet.size());
   Set messageIds = new HashSet<>();
   oldOpenSet.forEach(messageIds::add);
   oldOpenSet.clear();
   consumerBase.redeliverUnacknowledgedMessages(messageIds);
   }
   toggle();< === toggle after timeout
   timeout = client.timer().newTimeout(this, ackTimeoutMillis, 
TimeUnit.MILLISECONDS);
   }
   }, ackTimeoutMillis, TimeUnit.MILLISECONDS);
   }
   ```
   before first timeout, all messageId was added in CurrentSet, not in 
OldOpenSet, so isAckTimeout() is false, and `redeliverUnacknowledgedMessages` 
was not called at first timeout.
   
   The fix may move `toggle()` from behind if clause to before if clause.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] dsambandam commented on issue #2576: Pulsar client consume command fails with java.lang.IllegalArgumentException: port out of range:-1

2018-09-14 Thread GitBox
dsambandam commented on issue #2576: Pulsar client consume command fails with 
java.lang.IllegalArgumentException: port out of range:-1
URL: 
https://github.com/apache/incubator-pulsar/issues/2576#issuecomment-421378520
 
 
   @sijie is this a bug as the same command runs fine with TLS disabled. Kindly 
confirm.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] dsambandam commented on issue #2576: Pulsar client consume command fails with java.lang.IllegalArgumentException: port out of range:-1

2018-09-14 Thread GitBox
dsambandam commented on issue #2576: Pulsar client consume command fails with 
java.lang.IllegalArgumentException: port out of range:-1
URL: 
https://github.com/apache/incubator-pulsar/issues/2576#issuecomment-421398295
 
 
   Stack Trace:
   java.lang.IllegalArgumentException: port out of range:-1
   
at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143) 
~[?:1.8.0_171]
   at 
java.net.InetSocketAddress.createUnresolved(InetSocketAddress.java:254) 
~[?:1.8.0_171]
   at 
org.apache.pulsar.client.impl.BinaryProtoLookupService.lambda$null$2(BinaryProtoLookupService.java:109)
 
~[org.apache.pulsar-pulsar-client-original-2.1.0-incubating.jar:2.1.0-incubating]
   at 
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) 
[?:1.8.0_171]
   at 
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
 [?:1.8.0_171]
   at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) 
[?:1.8.0_171]
   at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) 
[?:1.8.0_171]
   at 
org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:369)
 
[org.apache.pulsar-pulsar-client-original-2.1.0-incubating.jar:2.1.0-incubating]
   at 
org.apache.pulsar.common.api.PulsarDecoder.channelRead(PulsarDecoder.java:115) 
[org.apache.pulsar-pulsar-common-2.1.0-incubating.jar:2.1.0-incubating]
   at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1389) 
[io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1171) 
[io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1205) 
[io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945)
 [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
   at 

[GitHub] sijie commented on issue #2584: unacked message is not redelivered in setting ackTimeout

2018-09-14 Thread GitBox
sijie commented on issue #2584: unacked message is not redelivered in setting 
ackTimeout
URL: 
https://github.com/apache/incubator-pulsar/issues/2584#issuecomment-421407995
 
 
   @jiazhai can you submit a PR for that?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] jiazhai commented on issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener

2018-09-14 Thread GitBox
jiazhai commented on issue #2574: Timeout message not get redeliver in 
TopicsConsumer when use message listener
URL: 
https://github.com/apache/incubator-pulsar/issues/2574#issuecomment-421381833
 
 
   The main reason for this error is that the message listener for 
TopicsConsumer was also set for individual sub-consumerImpl.
   In above calling stack, when time-out, TopicsConsumer.UnAckedMessageTracker 
will let sub-consumerImpl to redeliver messages.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[incubator-pulsar] branch master updated: Issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener

2018-09-14 Thread sijie
This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new ec74355  Issue #2574: Timeout message not get redeliver in 
TopicsConsumer when use message listener
ec74355 is described below

commit ec7435565520ee6ac3ed2ce4a0cd13e8de7dbca3
Author: Jia Zhai 
AuthorDate: Sat Sep 15 00:15:13 2018 +0800

Issue #2574: Timeout message not get redeliver in TopicsConsumer when use 
message listener

### Motivation

fix issue #2574 .
Timeout message not get redeliver in TopicsConsumer when use message 
listener.
This is caused by message listener wrongly set in individual 
sub-ConsumerImpl.

### Modifications

set message listener to null for individual sub-ConsumerImpl.
Add a UT

### Result

UT passed.
---
 .../pulsar/client/impl/TopicsConsumerImplTest.java | 58 ++
 .../client/impl/MultiTopicsConsumerImpl.java   |  1 +
 2 files changed, 59 insertions(+)

diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
index d28d1d5..55be9ec 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
@@ -19,6 +19,7 @@
 package org.apache.pulsar.client.impl;
 
 import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
 import static org.testng.Assert.assertTrue;
 import static org.testng.Assert.fail;
 
@@ -615,4 +616,61 @@ public class TopicsConsumerImplTest extends 
ProducerConsumerBase {
 }
 }
 
+/**
+ * Test Listener for github issue #2547
+ */
+@Test(timeOut = 3)
+public void testMultiTopicsMessageListener() throws Exception {
+String key = "MultiTopicsMessageListenerTest";
+final String subscriptionName = "my-ex-subscription-" + key;
+final String messagePredicate = "my-message-" + key + "-";
+final int totalMessages = 6;
+
+// set latch larger than totalMessages, so timeout message get resend
+CountDownLatch latch = new CountDownLatch(totalMessages * 3);
+
+final String topicName1 = "persistent://prop/use/ns-abc/topic-1-" + 
key;
+List topicNames = Lists.newArrayList(topicName1);
+
+admin.tenants().createTenant("prop", new TenantInfo());
+admin.topics().createPartitionedTopic(topicName1, 2);
+
+// 1. producer connect
+Producer producer1 = 
pulsarClient.newProducer().topic(topicName1)
+.enableBatching(false)
+.messageRoutingMode(MessageRoutingMode.SinglePartition)
+.create();
+
+// 2. Create consumer, set not ack in message listener, so time-out 
message will resend
+Consumer consumer = pulsarClient.newConsumer()
+.topics(topicNames)
+.subscriptionName(subscriptionName)
+.subscriptionType(SubscriptionType.Shared)
+.ackTimeout(1000, TimeUnit.MILLISECONDS)
+.receiverQueueSize(100)
+.messageListener((c1, msg) -> {
+assertNotNull(msg, "Message cannot be null");
+String receivedMessage = new String(msg.getData());
+latch.countDown();
+
+log.info("Received message [{}] in the listener, latch: {}",
+receivedMessage, latch.getCount());
+// since not acked, it should retry another time
+//c1.acknowledgeAsync(msg);
+})
+.subscribe();
+assertTrue(consumer instanceof MultiTopicsConsumerImpl);
+
+MultiTopicsConsumerImpl topicsConsumer = (MultiTopicsConsumerImpl) 
consumer;
+
+// 3. producer publish messages
+for (int i = 0; i < totalMessages; i++) {
+producer1.send((messagePredicate + "producer1-" + i).getBytes());
+}
+
+// verify should not time out, because of message redelivered several 
times.
+latch.await();
+
+consumer.close();
+}
 }
diff --git 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
index 75fdac6..8b8556a 100644
--- 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
+++ 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
@@ -485,6 +485,7 @@ public class MultiTopicsConsumerImpl extends 
ConsumerBase {
 ConsumerConfigurationData internalConsumerConfig = conf.clone();
 internalConsumerConfig.setSubscriptionName(subscription);
 

[GitHub] sijie closed pull request #2583: Issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener

2018-09-14 Thread GitBox
sijie closed pull request #2583: Issue #2574: Timeout message not get redeliver 
in TopicsConsumer when use message listener 
URL: https://github.com/apache/incubator-pulsar/pull/2583
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
index d28d1d5f33..55be9ecbd4 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
@@ -19,6 +19,7 @@
 package org.apache.pulsar.client.impl;
 
 import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
 import static org.testng.Assert.assertTrue;
 import static org.testng.Assert.fail;
 
@@ -615,4 +616,61 @@ public void testTopicsNameSubscribeWithBuilderFail() 
throws Exception {
 }
 }
 
+/**
+ * Test Listener for github issue #2547
+ */
+@Test(timeOut = 3)
+public void testMultiTopicsMessageListener() throws Exception {
+String key = "MultiTopicsMessageListenerTest";
+final String subscriptionName = "my-ex-subscription-" + key;
+final String messagePredicate = "my-message-" + key + "-";
+final int totalMessages = 6;
+
+// set latch larger than totalMessages, so timeout message get resend
+CountDownLatch latch = new CountDownLatch(totalMessages * 3);
+
+final String topicName1 = "persistent://prop/use/ns-abc/topic-1-" + 
key;
+List topicNames = Lists.newArrayList(topicName1);
+
+admin.tenants().createTenant("prop", new TenantInfo());
+admin.topics().createPartitionedTopic(topicName1, 2);
+
+// 1. producer connect
+Producer producer1 = 
pulsarClient.newProducer().topic(topicName1)
+.enableBatching(false)
+.messageRoutingMode(MessageRoutingMode.SinglePartition)
+.create();
+
+// 2. Create consumer, set not ack in message listener, so time-out 
message will resend
+Consumer consumer = pulsarClient.newConsumer()
+.topics(topicNames)
+.subscriptionName(subscriptionName)
+.subscriptionType(SubscriptionType.Shared)
+.ackTimeout(1000, TimeUnit.MILLISECONDS)
+.receiverQueueSize(100)
+.messageListener((c1, msg) -> {
+assertNotNull(msg, "Message cannot be null");
+String receivedMessage = new String(msg.getData());
+latch.countDown();
+
+log.info("Received message [{}] in the listener, latch: {}",
+receivedMessage, latch.getCount());
+// since not acked, it should retry another time
+//c1.acknowledgeAsync(msg);
+})
+.subscribe();
+assertTrue(consumer instanceof MultiTopicsConsumerImpl);
+
+MultiTopicsConsumerImpl topicsConsumer = (MultiTopicsConsumerImpl) 
consumer;
+
+// 3. producer publish messages
+for (int i = 0; i < totalMessages; i++) {
+producer1.send((messagePredicate + "producer1-" + i).getBytes());
+}
+
+// verify should not time out, because of message redelivered several 
times.
+latch.await();
+
+consumer.close();
+}
 }
diff --git 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
index 75fdac681a..8b8556adaf 100644
--- 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
+++ 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
@@ -485,6 +485,7 @@ String getHandlerName() {
 ConsumerConfigurationData internalConsumerConfig = conf.clone();
 internalConsumerConfig.setSubscriptionName(subscription);
 internalConsumerConfig.setConsumerName(consumerName);
+internalConsumerConfig.setMessageListener(null);
 return internalConsumerConfig;
 }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] srkukarni commented on issue #2577: fix behavior of JSONSchema for derived classes

2018-09-14 Thread GitBox
srkukarni commented on issue #2577: fix behavior of JSONSchema for derived 
classes
URL: https://github.com/apache/incubator-pulsar/pull/2577#issuecomment-421415097
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] srkukarni commented on issue #2579: Misc fixes

2018-09-14 Thread GitBox
srkukarni commented on issue #2579: Misc fixes
URL: https://github.com/apache/incubator-pulsar/pull/2579#issuecomment-421415207
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] dsambandam edited a comment on issue #2576: Pulsar client consume command fails with java.lang.IllegalArgumentException: port out of range:-1

2018-09-14 Thread GitBox
dsambandam edited a comment on issue #2576: Pulsar client consume command fails 
with java.lang.IllegalArgumentException: port out of range:-1
URL: 
https://github.com/apache/incubator-pulsar/issues/2576#issuecomment-421435649
 
 
   ./bin/pulsar initialize-cluster-metadata --cluster cluster1\
   --zookeeper zk1:2181 \
   --configuration-store zk1:2184 \
   --web-service-url http://broker1:8080 \
   --web-service-url-tls https://broker1:8443 \
   --broker-service-url pulsar://broker1:6650 \
   --broker-service-url-tls pulsar+ssl://broker1:6651
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2503: add auto ack and timeout configurable

2018-09-14 Thread GitBox
sijie commented on issue #2503: add auto ack and timeout configurable
URL: https://github.com/apache/incubator-pulsar/pull/2503#issuecomment-421431340
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] srkukarni commented on issue #2577: fix behavior of JSONSchema for derived classes

2018-09-14 Thread GitBox
srkukarni commented on issue #2577: fix behavior of JSONSchema for derived 
classes
URL: https://github.com/apache/incubator-pulsar/pull/2577#issuecomment-421432063
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] rdhabalia commented on issue #2535: Add ledger op timeout to avoid topics stuck on ledger-creation

2018-09-14 Thread GitBox
rdhabalia commented on issue #2535: Add ledger op timeout to avoid topics stuck 
on ledger-creation
URL: https://github.com/apache/incubator-pulsar/pull/2535#issuecomment-421462070
 
 
   rerun integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] jerrypeng commented on issue #2577: fix behavior of JSONSchema for derived classes

2018-09-14 Thread GitBox
jerrypeng commented on issue #2577: fix behavior of JSONSchema for derived 
classes
URL: https://github.com/apache/incubator-pulsar/pull/2577#issuecomment-421432872
 
 
   rerun integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie opened a new pull request #2585: [tests] Make BrokerClientIntegrationTest testing behavior deterministic

2018-09-14 Thread GitBox
sijie opened a new pull request #2585: [tests] Make BrokerClientIntegrationTest 
testing behavior deterministic
URL: https://github.com/apache/incubator-pulsar/pull/2585
 
 
   
   *Motivation*
   
   The test is flaky.
   
   ```
   2018-09-14\T\17:46:29.848 [ERROR] 
testUnsupportedBatchMessageConsumer(org.apache.pulsar.client.impl.BrokerClientIntegrationTest)
  Time elapsed: 3.161 s  <<< FAILURE!
   java.lang.AssertionError: Received message my-message-5 did not match the 
expected message my-message-0 expected [my-message-0] but found [my-message-5]
   at org.testng.Assert.fail(Assert.java:96)
   at org.testng.Assert.failNotEquals(Assert.java:776)
   at org.testng.Assert.assertEqualsImpl(Assert.java:137)
   at org.testng.Assert.assertEquals(Assert.java:118)
   at 
org.apache.pulsar.client.api.ProducerConsumerBase.testMessageOrderAndDuplicates(ProducerConsumerBase.java:51)
   at 
org.apache.pulsar.client.impl.BrokerClientIntegrationTest.testUnsupportedBatchMessageConsumer(BrokerClientIntegrationTest.java:357)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at 
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
   at 
org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
   at 
org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
   at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
   ```
   
   The problem is due to we used a `time` based batching policy in testing. 
There is no guarantee how messages can be batched, hence
   the ordering and duplication check can fail on shared subscription
   
   *Changes*
   
   Set batching delay to a very large value and make sure the messages are in 
one batch.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2500: [schema] add schemas for primtive types

2018-09-14 Thread GitBox
sijie commented on issue #2500: [schema] add schemas for primtive types
URL: https://github.com/apache/incubator-pulsar/pull/2500#issuecomment-421429724
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] dsambandam commented on issue #2576: Pulsar client consume command fails with java.lang.IllegalArgumentException: port out of range:-1

2018-09-14 Thread GitBox
dsambandam commented on issue #2576: Pulsar client consume command fails with 
java.lang.IllegalArgumentException: port out of range:-1
URL: 
https://github.com/apache/incubator-pulsar/issues/2576#issuecomment-421435649
 
 
   ./bin/pulsar initialize-cluster-metadata --cluster cluster1\
   --zookeeper zk1:2181 \
   --configuration-store zk1:2184 \
   --web-service-url broker1:8080 \
   --web-service-url-tls broker1:8443 \
   --broker-service-url pulsar://broker1:6650 \
   --broker-service-url-tls pulsar+ssl://broker1:6651
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] srkukarni commented on issue #2579: Misc fixes

2018-09-14 Thread GitBox
srkukarni commented on issue #2579: Misc fixes
URL: https://github.com/apache/incubator-pulsar/pull/2579#issuecomment-421463863
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] srkukarni commented on issue #2579: Misc fixes

2018-09-14 Thread GitBox
srkukarni commented on issue #2579: Misc fixes
URL: https://github.com/apache/incubator-pulsar/pull/2579#issuecomment-421510372
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] srkukarni opened a new pull request #2588: Preserve filename of the original function submission

2018-09-14 Thread GitBox
srkukarni opened a new pull request #2588: Preserve filename of the original 
function submission
URL: https://github.com/apache/incubator-pulsar/pull/2588
 
 
   ### Motivation
   
   When we submit a function, currently we don't note down the original 
filename that was uploaded from the client. For regular java functions this 
works fine. For python workers we manage by doing tricks wrt tricks like 
filename using classnames. However when we add things like wheel support, the 
filename encodes information that cannot be discarded away.
   This pr preserves that filename
   
   ### Modifications
   
   Describe the modifications you've done.
   
   ### Result
   
   After your change, what will change.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] aahmed-se commented on issue #2578: [WIP] Add support for schema extraction from a jar

2018-09-14 Thread GitBox
aahmed-se commented on issue #2578: [WIP] Add support for schema extraction 
from a jar
URL: https://github.com/apache/incubator-pulsar/pull/2578#issuecomment-421532376
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] aahmed-se commented on issue #2504: Add Presto Sql Test

2018-09-14 Thread GitBox
aahmed-se commented on issue #2504: Add Presto Sql Test
URL: https://github.com/apache/incubator-pulsar/pull/2504#issuecomment-421532383
 
 
   run java8 tests
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2587: [tests] improve connector related integration tests

2018-09-14 Thread GitBox
sijie commented on issue #2587: [tests] improve connector related integration 
tests
URL: https://github.com/apache/incubator-pulsar/pull/2587#issuecomment-421532881
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2588: Preserve filename of the original function submission

2018-09-14 Thread GitBox
sijie commented on issue #2588: Preserve filename of the original function 
submission
URL: https://github.com/apache/incubator-pulsar/pull/2588#issuecomment-421532863
 
 
   There are two failures related to this PR. please take a look.
   
   ```
   
org.apache.pulsar.functions.worker.rest.api.v2.FunctionApiV2ResourceTest.testRegisterFunctionFileUrlWithValidSinkClass
   
org.apache.pulsar.functions.worker.rest.api.v2.FunctionApiV2ResourceTest.testUpdateFunctionWithUrl
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2585: [tests] Make BrokerClientIntegrationTest testing behavior deterministic

2018-09-14 Thread GitBox
sijie commented on issue #2585: [tests] Make BrokerClientIntegrationTest 
testing behavior deterministic
URL: https://github.com/apache/incubator-pulsar/pull/2585#issuecomment-421532954
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2586: fixing/improving logging for function instance

2018-09-14 Thread GitBox
sijie commented on issue #2586: fixing/improving logging for function instance
URL: https://github.com/apache/incubator-pulsar/pull/2586#issuecomment-421532919
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] pablichjenkov closed issue #2581: Android and IOS client support

2018-09-14 Thread GitBox
pablichjenkov closed issue #2581: Android and IOS client support
URL: https://github.com/apache/incubator-pulsar/issues/2581
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2577: fix behavior of JSONSchema for derived classes

2018-09-14 Thread GitBox
sijie commented on issue #2577: fix behavior of JSONSchema for derived classes
URL: https://github.com/apache/incubator-pulsar/pull/2577#issuecomment-421533014
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2572: Shorten the timeout value of C++ ZTS client

2018-09-14 Thread GitBox
sijie commented on issue #2572: Shorten the timeout value of C++ ZTS client
URL: https://github.com/apache/incubator-pulsar/pull/2572#issuecomment-421533045
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] aahmed-se commented on issue #2571: Change repository tag for test image

2018-09-14 Thread GitBox
aahmed-se commented on issue #2571: Change repository tag for test image
URL: https://github.com/apache/incubator-pulsar/pull/2571#issuecomment-421532365
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[incubator-pulsar] branch master updated: Misc fixes (#2579)

2018-09-14 Thread sijie
This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 858648e  Misc fixes (#2579)
858648e is described below

commit 858648eb24aeba9d95a7e83513830afd40cb21fe
Author: Sanjeev Kulkarni 
AuthorDate: Fri Sep 14 22:33:03 2018 -0700

Misc fixes (#2579)

This pr does a couple of minor nits
1. Add a log statement in Sink to make it consistent with the Source.
2. Added AllArgs/NoArgs constructor for some test related objects
---
 .../src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java| 2 ++
 .../org/apache/pulsar/functions/api/examples/CustomBaseObject.java| 2 ++
 .../org/apache/pulsar/functions/api/examples/CustomDerivedObject.java | 4 
 3 files changed, 8 insertions(+)

diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
index 835e288..4449045 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
@@ -205,6 +205,8 @@ public class PulsarSink implements Sink {
 
 @Override
 public void open(Map config, SinkContext sinkContext) 
throws Exception {
+log.info("Opening pulsar sink with config: {}", pulsarSinkConfig);
+
 Schema schema = initializeSchema();
 
 FunctionConfig.ProcessingGuarantees processingGuarantees = 
this.pulsarSinkConfig.getProcessingGuarantees();
diff --git 
a/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomBaseObject.java
 
b/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomBaseObject.java
index 173393b..6102887 100644
--- 
a/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomBaseObject.java
+++ 
b/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomBaseObject.java
@@ -20,11 +20,13 @@ package org.apache.pulsar.functions.api.examples;
 
 import lombok.AllArgsConstructor;
 import lombok.Getter;
+import lombok.NoArgsConstructor;
 import lombok.Setter;
 
 @Getter
 @Setter
 @AllArgsConstructor
+@NoArgsConstructor
 public class CustomBaseObject {
 private long baseValue;
 }
diff --git 
a/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomDerivedObject.java
 
b/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomDerivedObject.java
index 464e977..d700231 100644
--- 
a/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomDerivedObject.java
+++ 
b/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomDerivedObject.java
@@ -18,11 +18,15 @@
  */
 package org.apache.pulsar.functions.api.examples;
 
+import lombok.AllArgsConstructor;
 import lombok.Getter;
+import lombok.NoArgsConstructor;
 import lombok.Setter;
 
 @Getter
 @Setter
+@AllArgsConstructor
+@NoArgsConstructor
 public class CustomDerivedObject extends CustomBaseObject {
 private int derivedValue;
 public CustomDerivedObject(long baseValue, int derivedValue) {



[GitHub] sijie closed pull request #2579: Misc fixes

2018-09-14 Thread GitBox
sijie closed pull request #2579: Misc fixes
URL: https://github.com/apache/incubator-pulsar/pull/2579
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
index 835e288075..44490454c3 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
@@ -205,6 +205,8 @@ public PulsarSink(PulsarClient client, PulsarSinkConfig 
pulsarSinkConfig, String
 
 @Override
 public void open(Map config, SinkContext sinkContext) 
throws Exception {
+log.info("Opening pulsar sink with config: {}", pulsarSinkConfig);
+
 Schema schema = initializeSchema();
 
 FunctionConfig.ProcessingGuarantees processingGuarantees = 
this.pulsarSinkConfig.getProcessingGuarantees();
diff --git 
a/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomBaseObject.java
 
b/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomBaseObject.java
index 173393b62a..61028879fe 100644
--- 
a/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomBaseObject.java
+++ 
b/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomBaseObject.java
@@ -20,11 +20,13 @@
 
 import lombok.AllArgsConstructor;
 import lombok.Getter;
+import lombok.NoArgsConstructor;
 import lombok.Setter;
 
 @Getter
 @Setter
 @AllArgsConstructor
+@NoArgsConstructor
 public class CustomBaseObject {
 private long baseValue;
 }
diff --git 
a/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomDerivedObject.java
 
b/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomDerivedObject.java
index 464e977be8..d70023133f 100644
--- 
a/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomDerivedObject.java
+++ 
b/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CustomDerivedObject.java
@@ -18,11 +18,15 @@
  */
 package org.apache.pulsar.functions.api.examples;
 
+import lombok.AllArgsConstructor;
 import lombok.Getter;
+import lombok.NoArgsConstructor;
 import lombok.Setter;
 
 @Getter
 @Setter
+@AllArgsConstructor
+@NoArgsConstructor
 public class CustomDerivedObject extends CustomBaseObject {
 private int derivedValue;
 public CustomDerivedObject(long baseValue, int derivedValue) {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2543: Add ServiceUrlProvider and add method forceCloseConnection in PulsarC…

2018-09-14 Thread GitBox
sijie commented on issue #2543: Add ServiceUrlProvider and add method 
forceCloseConnection in PulsarC…
URL: https://github.com/apache/incubator-pulsar/pull/2543#issuecomment-421533108
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2500: [schema] add schemas for primtive types

2018-09-14 Thread GitBox
sijie commented on issue #2500: [schema] add schemas for primtive types
URL: https://github.com/apache/incubator-pulsar/pull/2500#issuecomment-421533163
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] jerrypeng opened a new pull request #2586: fixing/improving logging for function instance

2018-09-14 Thread GitBox
jerrypeng opened a new pull request #2586: fixing/improving logging for 
function instance
URL: https://github.com/apache/incubator-pulsar/pull/2586
 
 
   ### Motivation
   logging for java function instance (process mode) is not working well.  Log 
files are created at inappropriate locations for logs not captured by the 
routing appender.  
   
   The best for logging is just to put all the logs from the java instance log 
in a single log and not worry about the routing which is only useful when 
running in threaded mode
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] srkukarni commented on issue #2579: Misc fixes

2018-09-14 Thread GitBox
srkukarni commented on issue #2579: Misc fixes
URL: https://github.com/apache/incubator-pulsar/pull/2579#issuecomment-421528509
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2585: [tests] Make BrokerClientIntegrationTest testing behavior deterministic

2018-09-14 Thread GitBox
sijie commented on issue #2585: [tests] Make BrokerClientIntegrationTest 
testing behavior deterministic
URL: https://github.com/apache/incubator-pulsar/pull/2585#issuecomment-421482952
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie opened a new pull request #2587: [tests] improve connector related integration tests

2018-09-14 Thread GitBox
sijie opened a new pull request #2587: [tests] improve connector related 
integration tests
URL: https://github.com/apache/incubator-pulsar/pull/2587
 
 
   *Motivation*
   
   with more and more connector are added, it becomes expensive to start all 
external services at the begin.
   
   *Changes*
   
   - refactor the connector testing framework to start external service before 
methods
   - fix kafka, cassandra and mysql connectors
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[incubator-pulsar] branch master updated: PIP-22: Dead Letter Topic (#2508)

2018-09-14 Thread sijie
This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 95fe84c  PIP-22: Dead Letter Topic (#2508)
95fe84c is described below

commit 95fe84c7b43060102bb08f0dfe85dadb152f4542
Author: penghui 
AuthorDate: Fri Sep 14 15:37:58 2018 +0800

PIP-22: Dead Letter Topic (#2508)

### Motivation

Fixes #189

When consumer got messages from pulsar, It's difficult to ensure every 
message can be consume success. Pulsar support message redelivery feature by 
set acknowledge timeout when create a new consumer. This is a good feature 
guarantee consumer will not lost messages.

But however, some message will redelivery so many times possible, even to 
the extent that it can be never stop.

So, It's necessary to support a feature to control it by pulsar. Users can 
use this feature and customize this feature to control the message redelivery 
behavior. The feature named Dead Letter Topic.

### Modifications

Consumer can set maximum number of redeliveries by java client.
Consumer can set the name of Dead Letter Topic by java client, It’s not 
necessary.
Message exceeding the maximum number of redeliveries should send to Dead 
Letter Topic and acknowledged automatic.

### Result

If consumer enable future of dead letter topic. When Message exceeding the 
maximum number of redeliveries, message will send to the Dead Letter Topic and 
acknowledged automatic.
---
 .../org/apache/pulsar/broker/service/Consumer.java |   3 +-
 .../apache/pulsar/broker/service/Dispatcher.java   |   2 +
 .../broker/service/InMemoryRedeliveryTracker.java  |  58 +
 .../pulsar/broker/service/RedeliveryTracker.java   |  36 +++
 .../broker/service/RedeliveryTrackerDisabled.java  |  55 +
 .../NonPersistentDispatcherMultipleConsumers.java  |   9 +
 ...onPersistentDispatcherSingleActiveConsumer.java |   9 +
 .../PersistentDispatcherMultipleConsumers.java |  14 +-
 .../PersistentDispatcherSingleActiveConsumer.java  |  11 +
 .../service/persistent/PersistentSubscription.java |   1 +
 .../apache/pulsar/client/impl/RawReaderImpl.java   |   2 +-
 .../pulsar/client/api/DeadLetterTopicTest.java | 249 +
 .../client/impl/CompactedOutBatchMessageTest.java  |   2 +-
 .../apache/pulsar/client/api/ConsumerBuilder.java  |  24 ++
 .../apache/pulsar/client/api/DeadLetterPolicy.java |  32 +++
 .../org/apache/pulsar/client/impl/ClientCnx.java   |   2 +-
 .../pulsar/client/impl/ConsumerBuilderImpl.java|   7 +
 .../apache/pulsar/client/impl/ConsumerImpl.java| 111 -
 .../impl/conf/ConsumerConfigurationData.java   |   3 +
 .../org/apache/pulsar/common/api/Commands.java |   5 +-
 .../apache/pulsar/common/api/proto/PulsarApi.java  |  57 +
 pulsar-common/src/main/proto/PulsarApi.proto   |   1 +
 22 files changed, 680 insertions(+), 13 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
index 883bf55..5752034 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
@@ -265,7 +265,8 @@ public class Consumer {
 if (i == (entries.size() - 1)) {
 promise = writePromise;
 }
-ctx.write(Commands.newMessage(consumerId, messageId, 
metadataAndPayload), promise);
+int redeliveryCount = 
subscription.getDispatcher().getRedeliveryTracker().getRedeliveryCount(PositionImpl.get(messageId.getLedgerId(),
 messageId.getEntryId()));
+ctx.write(Commands.newMessage(consumerId, messageId, 
redeliveryCount, metadataAndPayload), promise);
 messageId.recycle();
 messageIdBuilder.recycle();
 entry.release();
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java
index 0d7b7d6..43f65cd 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java
@@ -70,4 +70,6 @@ public interface Dispatcher {
 
 void addUnAckedMessages(int unAckMessages);
 
+RedeliveryTracker getRedeliveryTracker();
+
 }
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/InMemoryRedeliveryTracker.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/InMemoryRedeliveryTracker.java
new file mode 100644
index 000..99b38cc
--- /dev/null
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/InMemoryRedeliveryTracker.java
@@ 

[GitHub] sijie commented on issue #2543: Add ServiceUrlProvider and add method forceCloseConnection in PulsarC…

2018-09-14 Thread GitBox
sijie commented on issue #2543: Add ServiceUrlProvider and add method 
forceCloseConnection in PulsarC…
URL: https://github.com/apache/incubator-pulsar/pull/2543#issuecomment-421259525
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie closed pull request #2508: PIP-22: Dead Letter Topic

2018-09-14 Thread GitBox
sijie closed pull request #2508: PIP-22: Dead Letter Topic
URL: https://github.com/apache/incubator-pulsar/pull/2508
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
index 883bf5553e..5752034a79 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
@@ -265,7 +265,8 @@ public SendMessageInfo sendMessages(final List 
entries, SendListener list
 if (i == (entries.size() - 1)) {
 promise = writePromise;
 }
-ctx.write(Commands.newMessage(consumerId, messageId, 
metadataAndPayload), promise);
+int redeliveryCount = 
subscription.getDispatcher().getRedeliveryTracker().getRedeliveryCount(PositionImpl.get(messageId.getLedgerId(),
 messageId.getEntryId()));
+ctx.write(Commands.newMessage(consumerId, messageId, 
redeliveryCount, metadataAndPayload), promise);
 messageId.recycle();
 messageIdBuilder.recycle();
 entry.release();
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java
index 0d7b7d6e55..43f65cd73b 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java
@@ -70,4 +70,6 @@
 
 void addUnAckedMessages(int unAckMessages);
 
+RedeliveryTracker getRedeliveryTracker();
+
 }
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/InMemoryRedeliveryTracker.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/InMemoryRedeliveryTracker.java
new file mode 100644
index 00..99b38ccf2f
--- /dev/null
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/InMemoryRedeliveryTracker.java
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import org.apache.bookkeeper.mledger.Position;
+
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class InMemoryRedeliveryTracker implements RedeliveryTracker {
+
+private ConcurrentHashMap trackerCache = new 
ConcurrentHashMap<>(16);
+
+@Override
+public int incrementAndGetRedeliveryCount(Position position) {
+trackerCache.putIfAbsent(position, new AtomicInteger(0));
+return trackerCache.get(position).incrementAndGet();
+}
+
+@Override
+public int getRedeliveryCount(Position position) {
+return trackerCache.getOrDefault(position, new AtomicInteger(0)).get();
+}
+
+@Override
+public void remove(Position position) {
+trackerCache.remove(position);
+}
+
+@Override
+public void removeBatch(List positions) {
+if (positions != null) {
+positions.forEach(this::remove);
+}
+}
+
+@Override
+public void clear() {
+trackerCache.clear();
+}
+}
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/RedeliveryTracker.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/RedeliveryTracker.java
new file mode 100644
index 00..0f2e54a542
--- /dev/null
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/RedeliveryTracker.java
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use 

[GitHub] sijie commented on issue #2508: PIP-22: Dead Letter Topic

2018-09-14 Thread GitBox
sijie commented on issue #2508: PIP-22: Dead Letter Topic
URL: https://github.com/apache/incubator-pulsar/pull/2508#issuecomment-421259811
 
 
   @codelipenghui it is a great contribution! thank you!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie closed pull request #2560: [proxy][functions] Issue #2154: proxy should be able to forward rest requests to function workers cluster

2018-09-14 Thread GitBox
sijie closed pull request #2560: [proxy][functions] Issue #2154: proxy should 
be able to forward rest requests to function workers cluster
URL: https://github.com/apache/incubator-pulsar/pull/2560
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/conf/proxy.conf b/conf/proxy.conf
index 9b307ccdc7..b95c4d314e 100644
--- a/conf/proxy.conf
+++ b/conf/proxy.conf
@@ -23,6 +23,11 @@ zookeeperServers=
 # Configuration store connection string (as a comma-separated list)
 configurationStoreServers=
 
+# If function workers are setup in a separate cluster, configure the following 
2 settings
+# to point to the function workers cluster
+functionWorkerWebServiceURL=
+functionWorkerWebServiceURLTLS=
+
 # ZooKeeper session timeout (in milliseconds)
 zookeeperSessionTimeoutMs=3
 
diff --git 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
index 49c789c6f7..d6c32df2ff 100644
--- 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
+++ 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
@@ -49,12 +49,15 @@
 private final ProxyConfiguration config;
 private final BrokerDiscoveryProvider discoveryProvider;
 private final String brokerWebServiceUrl;
+private final String functionWorkerWebServiceUrl;
 
 AdminProxyHandler(ProxyConfiguration config, BrokerDiscoveryProvider 
discoveryProvider) {
 this.config = config;
 this.discoveryProvider = discoveryProvider;
 this.brokerWebServiceUrl = config.isTlsEnabledWithBroker() ? 
config.getBrokerWebServiceURLTLS()
 : config.getBrokerWebServiceURL();
+this.functionWorkerWebServiceUrl = config.isTlsEnabledWithBroker() ? 
config.getFunctionWorkerWebServiceURLTLS()
+: config.getFunctionWorkerWebServiceURL();
 }
 
 @Override
@@ -122,7 +125,16 @@ protected HttpClient newHttpClient() {
 protected String rewriteTarget(HttpServletRequest request) {
 StringBuilder url = new StringBuilder();
 
-if (isBlank(brokerWebServiceUrl)) {
+boolean isFunctionsRestRequest = false;
+String requestUri = request.getRequestURI();
+if (requestUri.startsWith("/admin/v2/functions")
+|| requestUri.startsWith("/admin/functions")) {
+isFunctionsRestRequest = true;
+}
+
+if (isFunctionsRestRequest && !isBlank(functionWorkerWebServiceUrl)) {
+url.append(functionWorkerWebServiceUrl);
+} else if (isBlank(brokerWebServiceUrl)) {
 try {
 ServiceLookupData availableBroker = 
discoveryProvider.nextBroker();
 
@@ -148,7 +160,7 @@ protected String rewriteTarget(HttpServletRequest request) {
 if (url.lastIndexOf("/") == url.length() - 1) {
 url.deleteCharAt(url.lastIndexOf("/"));
 }
-url.append(request.getRequestURI());
+url.append(requestUri);
 
 String query = request.getQueryString();
 if (query != null) {
diff --git 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
index b4e8afb7c2..0155baad7b 100644
--- 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
+++ 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
@@ -48,6 +48,10 @@
 private String brokerWebServiceURL;
 private String brokerWebServiceURLTLS;
 
+// function worker web services
+private String functionWorkerWebServiceURL;
+private String functionWorkerWebServiceURLTLS;
+
 // Port to use to server binary-proto request
 private int servicePort = 6650;
 // Port to use to server binary-proto-tls request
@@ -158,6 +162,14 @@ public void setBrokerWebServiceURLTLS(String 
brokerWebServiceURLTLS) {
 this.brokerWebServiceURLTLS = brokerWebServiceURLTLS;
 }
 
+public String getFunctionWorkerWebServiceURL() {
+return functionWorkerWebServiceURL;
+}
+
+public String getFunctionWorkerWebServiceURLTLS() {
+return functionWorkerWebServiceURLTLS;
+}
+
 public String getZookeeperServers() {
 return zookeeperServers;
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie closed issue #2154: How pulsar proxy routes http requests to function workers

2018-09-14 Thread GitBox
sijie closed issue #2154: How pulsar proxy routes http requests to function 
workers
URL: https://github.com/apache/incubator-pulsar/issues/2154
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #2571: Change repository tag for test image

2018-09-14 Thread GitBox
sijie commented on issue #2571: Change repository tag for test image
URL: https://github.com/apache/incubator-pulsar/pull/2571#issuecomment-421267694
 
 
   run java8 tests
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[incubator-pulsar] branch master updated: [proxy][functions] Issue #2154: proxy should be able to forward rest requests to function workers cluster (#2560)

2018-09-14 Thread sijie
This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 07d4226  [proxy][functions] Issue #2154: proxy should be able to 
forward rest requests to function workers cluster (#2560)
07d4226 is described below

commit 07d42261eb2d685de6f9a5ab214f4d08944ffb6e
Author: Sijie Guo 
AuthorDate: Fri Sep 14 01:10:35 2018 -0700

[proxy][functions] Issue #2154: proxy should be able to forward rest 
requests to function workers cluster (#2560)

*Motivation*

Function workers can be deployed as a separate cluster. If so, proxy is not 
able to forward the functions
related rest calls to the correct server.

*Changes*

Add two settings in proxy configuration to allow proxy configuring 
forwarding functions related rest calls
to function worker cluster.

*Tests*

Verified with changes in integration tests (manually). It is hard to add 
the integration tests based on
current integration tests. will add them in a separate PR.
---
 conf/proxy.conf  |  5 +
 .../apache/pulsar/proxy/server/AdminProxyHandler.java| 16 ++--
 .../apache/pulsar/proxy/server/ProxyConfiguration.java   | 12 
 3 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/conf/proxy.conf b/conf/proxy.conf
index 9b307cc..b95c4d3 100644
--- a/conf/proxy.conf
+++ b/conf/proxy.conf
@@ -23,6 +23,11 @@ zookeeperServers=
 # Configuration store connection string (as a comma-separated list)
 configurationStoreServers=
 
+# If function workers are setup in a separate cluster, configure the following 
2 settings
+# to point to the function workers cluster
+functionWorkerWebServiceURL=
+functionWorkerWebServiceURLTLS=
+
 # ZooKeeper session timeout (in milliseconds)
 zookeeperSessionTimeoutMs=3
 
diff --git 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
index 49c789c..d6c32df 100644
--- 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
+++ 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
@@ -49,12 +49,15 @@ class AdminProxyHandler extends AsyncProxyServlet {
 private final ProxyConfiguration config;
 private final BrokerDiscoveryProvider discoveryProvider;
 private final String brokerWebServiceUrl;
+private final String functionWorkerWebServiceUrl;
 
 AdminProxyHandler(ProxyConfiguration config, BrokerDiscoveryProvider 
discoveryProvider) {
 this.config = config;
 this.discoveryProvider = discoveryProvider;
 this.brokerWebServiceUrl = config.isTlsEnabledWithBroker() ? 
config.getBrokerWebServiceURLTLS()
 : config.getBrokerWebServiceURL();
+this.functionWorkerWebServiceUrl = config.isTlsEnabledWithBroker() ? 
config.getFunctionWorkerWebServiceURLTLS()
+: config.getFunctionWorkerWebServiceURL();
 }
 
 @Override
@@ -122,7 +125,16 @@ class AdminProxyHandler extends AsyncProxyServlet {
 protected String rewriteTarget(HttpServletRequest request) {
 StringBuilder url = new StringBuilder();
 
-if (isBlank(brokerWebServiceUrl)) {
+boolean isFunctionsRestRequest = false;
+String requestUri = request.getRequestURI();
+if (requestUri.startsWith("/admin/v2/functions")
+|| requestUri.startsWith("/admin/functions")) {
+isFunctionsRestRequest = true;
+}
+
+if (isFunctionsRestRequest && !isBlank(functionWorkerWebServiceUrl)) {
+url.append(functionWorkerWebServiceUrl);
+} else if (isBlank(brokerWebServiceUrl)) {
 try {
 ServiceLookupData availableBroker = 
discoveryProvider.nextBroker();
 
@@ -148,7 +160,7 @@ class AdminProxyHandler extends AsyncProxyServlet {
 if (url.lastIndexOf("/") == url.length() - 1) {
 url.deleteCharAt(url.lastIndexOf("/"));
 }
-url.append(request.getRequestURI());
+url.append(requestUri);
 
 String query = request.getQueryString();
 if (query != null) {
diff --git 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
index b4e8afb..0155baa 100644
--- 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
+++ 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
@@ -48,6 +48,10 @@ public class ProxyConfiguration implements 
PulsarConfiguration {
 private String brokerWebServiceURL;
 private String brokerWebServiceURLTLS;
 
+// function worker web services
+private 

[GitHub] sijie commented on issue #2579: Misc fixes

2018-09-14 Thread GitBox
sijie commented on issue #2579: Misc fixes
URL: https://github.com/apache/incubator-pulsar/pull/2579#issuecomment-421267425
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[incubator-pulsar] branch asf-site updated: Updated site at revision 95fe84c

2018-09-14 Thread mmerli
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new a72ff25  Updated site at revision 95fe84c
a72ff25 is described below

commit a72ff25292b9782961d55e1d92ab6952a8cdfdde
Author: jenkins 
AuthorDate: Fri Sep 14 08:43:21 2018 +

Updated site at revision 95fe84c
---
 content/api/client/allclasses-frame.html   |   1 +
 content/api/client/allclasses-noframe.html |   1 +
 content/api/client/index-all.html  |  10 ++
 .../apache/pulsar/client/api/ConsumerBuilder.html  |  77 +++
 .../apache/pulsar/client/api/CryptoKeyReader.html  |   4 +-
 ...ParameterSupport.html => DeadLetterPolicy.html} | 104 +++--
 .../api/EncodedAuthenticationParameterSupport.html |   4 +-
 .../apache/pulsar/client/api/package-frame.html|   1 +
 .../apache/pulsar/client/api/package-summary.html  |   8 +-
 .../org/apache/pulsar/client/api/package-tree.html |   1 +
 content/api/client/overview-tree.html  |   1 +
 .../docs/latest/adaptors/PulsarSpark/index.html|   2 +-
 .../docs/latest/adaptors/PulsarStorm/index.html|   2 +-
 content/docs/latest/admin-api/overview/index.html  |   8 +-
 content/docs/latest/clients/Java/index.html|   2 +-
 .../latest/cookbooks/PartitionedTopics/index.html  |   2 +-
 .../latest/cookbooks/RetentionExpiry/index.html|   2 +-
 .../docs/latest/cookbooks/compaction/index.html|   2 +-
 .../cookbooks/message-deduplication/index.html |   2 +-
 .../docs/latest/cookbooks/message-queue/index.html |   2 +-
 .../latest/cookbooks/tiered-storage/index.html |   2 +-
 .../docs/latest/deployment/aws-cluster/index.html  |   2 +-
 content/docs/latest/deployment/cluster/index.html  |   2 +-
 content/docs/latest/deployment/instance/index.html |   2 +-
 .../docs/latest/project/BinaryProtocol/index.html  |  10 ++
 content/docs/latest/reference/CliTools/index.html  |   4 +-
 .../docs/latest/security/authorization/index.html  |   4 +-
 content/ja/project/BinaryProtocol/index.html   |  10 ++
 content/swagger/swagger.json   |  68 +++---
 29 files changed, 212 insertions(+), 128 deletions(-)

diff --git a/content/api/client/allclasses-frame.html 
b/content/api/client/allclasses-frame.html
index dee5dcd..3244593 100644
--- a/content/api/client/allclasses-frame.html
+++ b/content/api/client/allclasses-frame.html
@@ -27,6 +27,7 @@
 ConsumerStats
 CryptoKeyReader
 DataURLStreamHandler
+DeadLetterPolicy
 EncodedAuthenticationParameterSupport
 EncryptionKeyInfo
 HashingScheme
diff --git a/content/api/client/allclasses-noframe.html 
b/content/api/client/allclasses-noframe.html
index 4154d55..cc8d392 100644
--- a/content/api/client/allclasses-noframe.html
+++ b/content/api/client/allclasses-noframe.html
@@ -27,6 +27,7 @@
 ConsumerStats
 CryptoKeyReader
 DataURLStreamHandler
+DeadLetterPolicy
 EncodedAuthenticationParameterSupport
 EncryptionKeyInfo
 HashingScheme
diff --git a/content/api/client/index-all.html 
b/content/api/client/index-all.html
index 2f68dc9..22b715e 100644
--- a/content/api/client/index-all.html
+++ b/content/api/client/index-all.html
@@ -490,6 +490,16 @@
 
 DataURLStreamHandler()
 - Constructor for class org.apache.pulsar.client.api.url.DataURLStreamHandler
 
+deadLetterPolicy(DeadLetterPolicy)
 - Method in interface org.apache.pulsar.client.api.ConsumerBuilder
+
+Set dead letter policy for consumer
+
+ By default some message will redelivery so many times possible, even to the 
extent that it can be never stop.
+
+DeadLetterPolicy - Class in org.apache.pulsar.client.api
+
+DeadLetterPolicy()
 - Constructor for class org.apache.pulsar.client.api.DeadLetterPolicy
+
 disableReplication()
 - Method in interface org.apache.pulsar.client.api.MessageBuilder
 
 Deprecated.
diff --git 
a/content/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html 
b/content/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html
index c3b8748..0ca529d 100644
--- a/content/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html
+++ b/content/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html
@@ -17,7 +17,7 @@
 catch(err) {
 }
 //-->
-var methods = 
{"i0":6,"i1":6,"i2":6,"i3":6,"i4":6,"i5":6,"i6":6,"i7":6,"i8":6,"i9":6,"i10":6,"i11":6,"i12":6,"i13":6,"i14":6,"i15":6,"i16":6,"i17":6,"i18":6,"i19":6,"i20":6,"i21":6,"i22":6,"i23":6,"i24":6,"i25":6};
+var methods = 
{"i0":6,"i1":6,"i2":6,"i3":6,"i4":6,"i5":6,"i6":6,"i7":6,"i8":6,"i9":6,"i10":6,"i11":6,"i12":6,"i13":6,"i14":6,"i15":6,"i16":6,"i17":6,"i18":6,"i19":6,"i20":6,"i21":6,"i22":6,"i23":6,"i24":6,"i25":6,"i26":6};
 var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],4:["t3","Abstract Methods"]};
 var altColor = "altColor";
 var rowColor = "rowColor";
@@ -176,116 +176,124 @@ extends Cloneable
 
 
 

[GitHub] ivankelly commented on issue #240: Intermittent test failures in ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors

2018-09-14 Thread GitBox
ivankelly commented on issue #240: Intermittent test failures in 
ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors
URL: 
https://github.com/apache/incubator-pulsar/issues/240#issuecomment-421279476
 
 
   Closed by #2573 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly closed issue #240: Intermittent test failures in ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors

2018-09-14 Thread GitBox
ivankelly closed issue #240: Intermittent test failures in 
ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors
URL: https://github.com/apache/incubator-pulsar/issues/240
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] jiazhai closed pull request #2575: Issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener

2018-09-14 Thread GitBox
jiazhai closed pull request #2575: Issue #2574: Timeout message not get 
redeliver in TopicsConsumer when use message listener
URL: https://github.com/apache/incubator-pulsar/pull/2575
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
index d28d1d5f33..b655842961 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java
@@ -19,6 +19,7 @@
 package org.apache.pulsar.client.impl;
 
 import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
 import static org.testng.Assert.assertTrue;
 import static org.testng.Assert.fail;
 
@@ -615,4 +616,61 @@ public void testTopicsNameSubscribeWithBuilderFail() 
throws Exception {
 }
 }
 
+/**
+ * Test Listener
+ */
+@Test(timeOut = 3)
+public void testMultiTopicsMessageListener() throws Exception {
+String key = "MultiTopicsMessageListenerTest";
+final String subscriptionName = "my-ex-subscription-" + key;
+final String messagePredicate = "my-message-" + key + "-";
+final int totalMessages = 6;
+
+// set latch larger than totalMessages, so timeout message get resend
+CountDownLatch latch = new CountDownLatch(totalMessages * 3);
+
+final String topicName1 = "persistent://prop/use/ns-abc/topic-1-" + 
key;
+List topicNames = Lists.newArrayList(topicName1);
+
+admin.tenants().createTenant("prop", new TenantInfo());
+admin.topics().createPartitionedTopic(topicName1, 2);
+
+// 1. producer connect
+Producer producer1 = 
pulsarClient.newProducer().topic(topicName1)
+.enableBatching(false)
+.messageRoutingMode(MessageRoutingMode.SinglePartition)
+.create();
+
+// 2. Create consumer, set not ack in message listener, so time-out 
message will resend
+Consumer consumer = pulsarClient.newConsumer()
+.topics(topicNames)
+.subscriptionName(subscriptionName)
+.subscriptionType(SubscriptionType.Shared)
+.ackTimeout(1000, TimeUnit.MILLISECONDS)
+.receiverQueueSize(4)
+.messageListener((c1, msg) -> {
+assertNotNull(msg, "Message cannot be null");
+String receivedMessage = new String(msg.getData());
+latch.countDown();
+
+log.info("Received message [{}] in the listener, latch: {}",
+receivedMessage, latch.getCount());
+// since not acked, it should retry another time
+//c1.acknowledgeAsync(msg);
+})
+.subscribe();
+assertTrue(consumer instanceof MultiTopicsConsumerImpl);
+
+MultiTopicsConsumerImpl topicsConsumer = (MultiTopicsConsumerImpl) 
consumer;
+
+// 3. producer publish messages
+for (int i = 0; i < totalMessages; i++) {
+producer1.send((messagePredicate + "producer1-" + i).getBytes());
+}
+
+// verify should not time out.
+latch.await();
+
+consumer.close();
+}
 }
diff --git 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
index a0a231902f..ced7fc2aed 100644
--- 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
+++ 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
@@ -981,12 +981,7 @@ protected synchronized void messageProcessed(Message 
msg) {
 if (id instanceof BatchMessageIdImpl) {
 id = new MessageIdImpl(id.getLedgerId(), id.getEntryId(), 
getPartitionIndex());
 }
-if (partitionIndex != -1) {
-// we should no longer track this message, TopicsConsumer will 
take care from now onwards
-unAckedMessageTracker.remove(id);
-} else {
-unAckedMessageTracker.add(id);
-}
+unAckedMessageTracker.add(id);
 }
 }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly opened a new issue #2582: Schema javadoc missing from client api docs

2018-09-14 Thread GitBox
ivankelly opened a new issue #2582: Schema javadoc missing from client api docs
URL: https://github.com/apache/incubator-pulsar/issues/2582
 
 
   http://pulsar.apache.org/api/client/ mentions passing a schema on 
constructor creating, but there's no linkage to JSONSchema or AVROSchema and 
the javadoc for these classes doesn't exist.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] jiazhai commented on issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener

2018-09-14 Thread GitBox
jiazhai commented on issue #2574: Timeout message not get redeliver in 
TopicsConsumer when use message listener
URL: 
https://github.com/apache/incubator-pulsar/issues/2574#issuecomment-421365678
 
 
   seems there was something wrong in above comments. will debug and post a new 
one. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services