[GitHub] srkukarni commented on issue #1520: Pulsar Connect

[GitBox]

srkukarni commented on issue #1520: Pulsar Connect
URL: https://github.com/apache/incubator-pulsar/pull/1520#issuecomment-379434630
 
 
   @merlimat I've addressed your comments. Please take a look again. Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] srkukarni commented on a change in pull request #1520: Pulsar Connect

[GitBox]

srkukarni commented on a change in pull request #1520: Pulsar Connect
URL: https://github.com/apache/incubator-pulsar/pull/1520#discussion_r179909351
 
 

 ##
 File path: pulsar-connect/twitter/pom.xml
 ##
 @@ -0,0 +1,49 @@
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  4.0.0
+  
+org.apache.pulsar
+pulsar-connect
+2.0.0-incubating-SNAPSHOT
+  
+
+  pulsar-connect-twitter
+  Pulsar Connect :: Twitter
+
+  
+
+
+  ${project.groupId}
+  pulsar-connect-core
+  ${project.version}
+
+
+
 
 Review comment:
   Added in top level pom


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat opened a new pull request #1523: Use signSafeMod in RoundRobinPartitionMessageRouter

[GitBox]

merlimat opened a new pull request #1523: Use signSafeMod in 
RoundRobinPartitionMessageRouter
URL: https://github.com/apache/incubator-pulsar/pull/1523
 
 
   ### Motivation
   
   I have seen this errors in master with the round-robin message router (used 
with `perf-producer`) : 
   
   ```
   00:11:08.789 [pulsar-perf-producer-exec-1-1] ERROR 
org.apache.pulsar.testclient.PerformanceProducer - Got error
   java.lang.IllegalArgumentException: Illegal partition index chosen by the 
message routing policy
   at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) 
~[com.google.guava-guava-20.0.jar:?]
   at 
org.apache.pulsar.client.impl.PartitionedProducerImpl.sendAsync(PartitionedProducerImpl.java:170)
 
~[org.apache.pulsar-pulsar-client-original-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]
   at 
org.apache.pulsar.client.impl.ProducerBase.sendAsync(ProducerBase.java:54) 
~[org.apache.pulsar-pulsar-client-original-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]
   at 
org.apache.pulsar.testclient.PerformanceProducer.lambda$0(PerformanceProducer.java:347)
 
~[org.apache.pulsar-pulsar-testclient-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]
   at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_161]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_161]
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_161]
   at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
   at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
   ```
   
   Fixed by consistently using `signSafeMod()` function. Imported from BK code 
since client doesn't depend on BK.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat opened a new pull request #1521: Removed contention between producers on ManagedLedger addEntry

[GitBox]

merlimat opened a new pull request #1521: Removed contention between producers 
on ManagedLedger addEntry
URL: https://github.com/apache/incubator-pulsar/pull/1521
 
 
   ### Motivation
   
   When there are multiple producers writing on the same topic and there are 
many entries/second (little batching), there can be thread contention when 
doing `managedLedger.addEntry()`. 
   
   ### Modifications
* Do the sync part of add entry from a thread hashed on the name of the 
managed ledger. 
   
   ### Result
   
   This will remove the contention on both the `ManagedLedger` and the BK 
`LedgerHandler` mutexes.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1520: Pulsar Connect

[GitBox]

merlimat commented on a change in pull request #1520: Pulsar Connect
URL: https://github.com/apache/incubator-pulsar/pull/1520#discussion_r179904692
 
 

 ##
 File path: 
pulsar-connect/twitter/src/main/java/org/apache/pulsar/connect/twitter/TwitterFireHoseConfigKeys.java
 ##
 @@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.connect.twitter;
+
+public class TwitterFireHoseConfigKeys {
+
+public static final String CONSUMER_KEY = "twitter-source.consumerKey";
 
 Review comment:
   I'm not sure `-` is valid character in a properties file. 
   
   Since we're starting from scratch though, we could also use yaml to 
configure the source/sinks and we could just define a Java pojo that will be 
filled from the yaml file. That will make it easier for implementations to have 
different types in the configuration.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1520: Pulsar Connect

[GitBox]

merlimat commented on a change in pull request #1520: Pulsar Connect
URL: https://github.com/apache/incubator-pulsar/pull/1520#discussion_r179904748
 
 

 ##
 File path: 
pulsar-connect/core/src/main/java/org/apache/pulsar/connect/core/Sink.java
 ##
 @@ -0,0 +1,46 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.connect.core;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A Pulsar Sink interface.
+ * The lifcycle is to open it, call publish to publish messages and
+ * then close it at the end of the session
+ */
+public interface Sink extends AutoCloseable {
+/**
+ * Open connector with configuration
+ *
+ * @param config initialization config
+ * @throws Exception IO type exceptions when opening a connector
+ */
+void open(final Map config) throws Exception;
+
+/**
+ * Attempt to publish a type safe collection of messages
+ *
+ * @param message Object to publish to the sink
+ * @return Completable future fo async publish request
+ */
+CompletableFuture publish(final T message);
 
 Review comment:
   If this sink is referred to "getting messages from a topic and write into a 
sink", then "publish" can probably a bit misleading


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1520: Pulsar Connect

[GitBox]

merlimat commented on a change in pull request #1520: Pulsar Connect
URL: https://github.com/apache/incubator-pulsar/pull/1520#discussion_r179904588
 
 

 ##
 File path: 
pulsar-connect/core/src/main/java/org/apache/pulsar/connect/core/Sink.java
 ##
 @@ -0,0 +1,46 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.connect.core;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A Pulsar Sink interface.
 
 Review comment:
   It is not immediately clear who is the "sink" here, whether this for reading 
*from* a topic or to write *into* a topic.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1520: Pulsar Connect

[GitBox]

merlimat commented on a change in pull request #1520: Pulsar Connect
URL: https://github.com/apache/incubator-pulsar/pull/1520#discussion_r179904923
 
 

 ##
 File path: pulsar-connect/twitter/pom.xml
 ##
 @@ -0,0 +1,49 @@
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  4.0.0
+  
+org.apache.pulsar
+pulsar-connect
+2.0.0-incubating-SNAPSHOT
+  
+
+  pulsar-connect-twitter
+  Pulsar Connect :: Twitter
+
+  
+
+
+  ${project.groupId}
+  pulsar-connect-core
+  ${project.version}
+
+
+
 
 Review comment:
   All versions should be added to dependencyManagement in top level pom, so 
that we can ensure we use 1 single version of a dependency across modules


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1520: Pulsar Connect

[GitBox]

merlimat commented on a change in pull request #1520: Pulsar Connect
URL: https://github.com/apache/incubator-pulsar/pull/1520#discussion_r179904898
 
 

 ##
 File path: pulsar-connect/pom.xml
 ##
 @@ -0,0 +1,59 @@
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  4.0.0
+  pom
+  
+org.apache.pulsar
+pulsar
+2.0.0-incubating-SNAPSHOT
+  
+
+  pulsar-connect
+  Pulsar Connect :: Parent
+
+  
+core
+twitter
+  
+
+  
 
 Review comment:
   Lombok is already defined in top-level pom and already injected in all 
submodules


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] srkukarni opened a new pull request #1520: Pulsar Connect

[GitBox]

srkukarni opened a new pull request #1520: Pulsar Connect
URL: https://github.com/apache/incubator-pulsar/pull/1520
 
 
   ### Motivation
   
   A significant barrier for adoption of Pulsar is the lack of connectors 
moving data in and out of Pulsar. This pr attempts to provide an interface for 
adding connectors to Pulsar. An example Twitter FireHose connector is included 
as well.
   
   ### Modifications
   
   Describe the modifications you've done.
   
   ### Result
   
   After your change, what will change.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1506: Underscore `_` should be allowed as property name

[GitBox]

merlimat commented on issue #1506: Underscore `_` should be allowed as property 
name
URL: 
https://github.com/apache/incubator-pulsar/issues/1506#issuecomment-379410968
 
 
   Added test case in #1516


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1516: Added test to ensure property name can contain '_' characters.

[GitBox]

merlimat closed pull request #1516: Added test to ensure property name can 
contain '_' characters.
URL: https://github.com/apache/incubator-pulsar/pull/1516
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java
index 2bc1652a78..c8c3b75804 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java
@@ -768,4 +768,41 @@ public void testPublishConsumerStats() throws Exception {
 producer.close();
 consumer.close();
 }
+
+@Test
+public void testTenantNameWithUnderscore() throws Exception {
+PropertyAdmin propertyAdmin = new 
PropertyAdmin(Sets.newHashSet("role1", "role2"), Sets.newHashSet("use"));
+admin.properties().createProperty("prop_xyz", propertyAdmin);
+
+admin.namespaces().createNamespace("prop_xyz/use/my-namespace");
+
+String topic = "persistent://prop_xyz/use/my-namespace/my-topic";
+
+Producer producer = pulsarClient.newProducer().topic(topic)
+.create();
+
+PersistentTopicStats stats = admin.persistentTopics().getStats(topic);
+assertEquals(stats.publishers.size(), 1);
+producer.close();
+}
+
+@Test
+public void testTenantNameWithInvalidCharacters() throws Exception {
+PropertyAdmin propertyAdmin = new 
PropertyAdmin(Sets.newHashSet("role1", "role2"), Sets.newHashSet("use"));
+
+// If we try to create property with invalid characters, it should 
fail immediately
+try {
+admin.properties().createProperty("prop xyz", propertyAdmin);
+fail("Should have failed");
+} catch (PulsarAdminException e) {
+// Expected
+}
+
+try {
+admin.properties().createProperty("prop", propertyAdmin);
+fail("Should have failed");
+} catch (PulsarAdminException e) {
+// Expected
+}
+}
 }
diff --git 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java 
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java
index 165fd07a9c..e5180d6935 100644
--- 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java
+++ 
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java
@@ -18,12 +18,14 @@
  */
 package org.apache.pulsar.common.naming;
 
-import java.net.URI;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
+import lombok.experimental.UtilityClass;
+
 /**
  */
+@UtilityClass
 public class NamedEntity {
 
 // allowed characters for property, namespace, cluster and topic names are
@@ -37,11 +39,4 @@ public static void checkName(String name) throws 
IllegalArgumentException {
 throw new IllegalArgumentException("Invalid named entity: " + 
name);
 }
 }
-
-public static void checkURI(URI uri, String name) {
-if (!String.format("%s://%s%s", uri.getScheme(), uri.getHost(), 
uri.getPath()).equals(name)) {
-throw new IllegalArgumentException("Invalid trailing chars in 
named entity: " + name);
-}
-}
-
 }
\ No newline at end of file
diff --git 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicDomain.java 
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicDomain.java
index 744efeaf28..26b382f52b 100644
--- 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicDomain.java
+++ 
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicDomain.java
@@ -30,14 +30,14 @@ private TopicDomain(String value) {
 public String value() {
 return this.value;
 }
-
+
 public static TopicDomain getEnum(String value) {
 for (TopicDomain e : values()) {
 if (e.value.equalsIgnoreCase(value)) {
 return e;
 }
 }
-throw new IllegalArgumentException("Invalid enum value " + value);
+throw new IllegalArgumentException("Invalid topic domain: '" + value + 
"'");
 }
 
 @Override
diff --git 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java 
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java
index 6bb2693286..aa7dd4ad86 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java
@@ -105,7 +105,7 @@ private TopicName(String completeTopicName) {
 // legacy: 

[incubator-pulsar] branch master updated: Added test to ensure property name can contain '_' characters. (#1516)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 3ae5115  Added test to ensure property name can contain '_' 
characters. (#1516)
3ae5115 is described below

commit 3ae511518558499729ea9ac2154b183718fc0668
Author: Matteo Merli 
AuthorDate: Fri Apr 6 16:33:10 2018 -0700

Added test to ensure property name can contain '_' characters. (#1516)
---
 .../apache/pulsar/broker/admin/AdminApiTest2.java  | 37 ++
 .../apache/pulsar/common/naming/NamedEntity.java   | 11 ++-
 .../apache/pulsar/common/naming/TopicDomain.java   |  4 +--
 .../org/apache/pulsar/common/naming/TopicName.java |  8 ++---
 4 files changed, 46 insertions(+), 14 deletions(-)

diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java
index 2bc1652..c8c3b75 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java
@@ -768,4 +768,41 @@ public class AdminApiTest2 extends 
MockedPulsarServiceBaseTest {
 producer.close();
 consumer.close();
 }
+
+@Test
+public void testTenantNameWithUnderscore() throws Exception {
+PropertyAdmin propertyAdmin = new 
PropertyAdmin(Sets.newHashSet("role1", "role2"), Sets.newHashSet("use"));
+admin.properties().createProperty("prop_xyz", propertyAdmin);
+
+admin.namespaces().createNamespace("prop_xyz/use/my-namespace");
+
+String topic = "persistent://prop_xyz/use/my-namespace/my-topic";
+
+Producer producer = pulsarClient.newProducer().topic(topic)
+.create();
+
+PersistentTopicStats stats = admin.persistentTopics().getStats(topic);
+assertEquals(stats.publishers.size(), 1);
+producer.close();
+}
+
+@Test
+public void testTenantNameWithInvalidCharacters() throws Exception {
+PropertyAdmin propertyAdmin = new 
PropertyAdmin(Sets.newHashSet("role1", "role2"), Sets.newHashSet("use"));
+
+// If we try to create property with invalid characters, it should 
fail immediately
+try {
+admin.properties().createProperty("prop xyz", propertyAdmin);
+fail("Should have failed");
+} catch (PulsarAdminException e) {
+// Expected
+}
+
+try {
+admin.properties().createProperty("prop", propertyAdmin);
+fail("Should have failed");
+} catch (PulsarAdminException e) {
+// Expected
+}
+}
 }
diff --git 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java 
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java
index 165fd07..e5180d6 100644
--- 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java
+++ 
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java
@@ -18,12 +18,14 @@
  */
 package org.apache.pulsar.common.naming;
 
-import java.net.URI;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
+import lombok.experimental.UtilityClass;
+
 /**
  */
+@UtilityClass
 public class NamedEntity {
 
 // allowed characters for property, namespace, cluster and topic names are
@@ -37,11 +39,4 @@ public class NamedEntity {
 throw new IllegalArgumentException("Invalid named entity: " + 
name);
 }
 }
-
-public static void checkURI(URI uri, String name) {
-if (!String.format("%s://%s%s", uri.getScheme(), uri.getHost(), 
uri.getPath()).equals(name)) {
-throw new IllegalArgumentException("Invalid trailing chars in 
named entity: " + name);
-}
-}
-
 }
\ No newline at end of file
diff --git 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicDomain.java 
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicDomain.java
index 744efea..26b382f 100644
--- 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicDomain.java
+++ 
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicDomain.java
@@ -30,14 +30,14 @@ public enum TopicDomain {
 public String value() {
 return this.value;
 }
-
+
 public static TopicDomain getEnum(String value) {
 for (TopicDomain e : values()) {
 if (e.value.equalsIgnoreCase(value)) {
 return e;
 }
 }
-throw new IllegalArgumentException("Invalid enum value " + value);
+throw new IllegalArgumentException("Invalid topic domain: '" + value + 
"'");
 }
 
 @Override
diff --git 
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java 

[GitHub] merlimat closed issue #1506: Underscore `_` should be allowed as property name

[GitBox]

merlimat closed issue #1506: Underscore `_` should be allowed as property name
URL: https://github.com/apache/incubator-pulsar/issues/1506
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1381: Schema registry 4/N

[GitBox]

merlimat closed pull request #1381: Schema registry 4/N
URL: https://github.com/apache/incubator-pulsar/pull/1381
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/SchemasResource.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/SchemasResource.java
new file mode 100644
index 00..2173617733
--- /dev/null
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/SchemasResource.java
@@ -0,0 +1,242 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.admin.v2;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.isNull;
+import static org.apache.commons.lang.StringUtils.defaultIfEmpty;
+import static org.apache.pulsar.common.util.Codec.decode;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
+import io.swagger.annotations.ApiOperation;
+import java.time.Clock;
+import java.util.Optional;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.DELETE;
+import javax.ws.rs.Encoded;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.container.AsyncResponse;
+import javax.ws.rs.container.Suspended;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.pulsar.broker.admin.AdminResource;
+import org.apache.pulsar.broker.service.Topic;
+import org.apache.pulsar.broker.web.RestException;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.schema.DeleteSchemaResponse;
+import org.apache.pulsar.common.schema.GetSchemaResponse;
+import org.apache.pulsar.common.schema.PostSchemaPayload;
+import org.apache.pulsar.common.schema.PostSchemaResponse;
+import org.apache.pulsar.common.schema.SchemaData;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.common.schema.SchemaVersion;
+
+@Path("/schemas")
+public class SchemasResource extends AdminResource {
+
+private final Clock clock;
+
+public SchemasResource() {
+this(Clock.systemUTC());
+}
+
+@VisibleForTesting
+public SchemasResource(Clock clock) {
+super();
+this.clock = clock;
+}
+
+@GET
+@Path("/{property}/{namespace}/{topic}/schema")
+@Produces(MediaType.APPLICATION_JSON)
+@ApiOperation(value = "Get topic schema", response = 
GetSchemaResponse.class)
+public void getSchema(
+@PathParam("property") String property,
+@PathParam("namespace") String namespace,
+@PathParam("topic") String topic,
+@Suspended final AsyncResponse response
+) {
+validateDestinationAndAdminOperation(property, namespace, topic);
+
+String schemaId = buildSchemaId(property, namespace, topic);
+pulsar().getSchemaRegistryService().getSchema(schemaId)
+.handle((schema, error) -> {
+if (isNull(error)) {
+response.resume(
+Response.ok()
+.encoding(MediaType.APPLICATION_JSON)
+.entity(GetSchemaResponse.builder()
+.version(schema.version)
+.type(schema.schema.getType())
+.timestamp(schema.schema.getTimestamp())
+.data(new String(schema.schema.getData()))
+.properties(schema.schema.getProps())
+.build()
+)
+.build()
+);
+} else {
+response.resume(error);
+}
+return null;
+});
+}
+
+@GET
+

[incubator-pulsar] branch master updated: Schema registry 4/N (#1381)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 1eb8068  Schema registry 4/N (#1381)
1eb8068 is described below

commit 1eb80681700a8dcf268f06b5f227033cd1165bb4
Author: Dave Rusek 
AuthorDate: Fri Apr 6 16:50:19 2018 -0600

Schema registry 4/N (#1381)

* Schema Registry proto changes

* Infrastructure to store schemas

* A default schema registry implementation

* Add admin api for the schema registry

* Renumber schema fields

* Update Pulsar API with schema changes

* Revert field number change

* Fix merge conflict

* Fix broken merge

* DestinationName has been renamed to TopicName

* Address issues in review

* Add schema type back to proto definition

* Address comments regarding lombok usage

* Remove reserved future enum fields

* regenerate code from protobuf

* Remove unused code

* Add schema version to producer success message

* plumb schema through to producer

* Revert "Add schema version to producer success message"

This reverts commit e7e72f468cf46f1605524a7399520c22763583c9.

* Revert "Revert "Add schema version to producer success message""

This reverts commit 7b902f6bdb1cb054e26577747ff4dd8c326a6248.

* Persist schema on producer connect

* Add principal to schema on publish

* Reformat function for readability

* Remove unused protoc profile

* Rename put on schema registry to putIfAbsent

* Reformat function for readability

* Remove unused protoc profile

* Rename put on schema registry to putIfAbsent

* fix compile errors from parent branch changes

* fix lombok tomfoolery on builder

* plumb hash through and allow lookup by data

* wip

* run tests

* wip: address review comments

* switch underscore to slash in schema name

* blah

* Get duplicate schema detection to work

* Fix protobuf version incompatibility

* fix merge issues

* Fix license headers

* Fix license headers

* Address review

* Fix webservice

* plumb schema from producer to server and back

* Plumb schema through subscriber

* Create and return schema via rest endpoint

* Make DELETE great again

* Clean up imports

* Move resource objects to common package

* Fix licenses

* Update error message for schema registry service

* Remove cruft

* Address review comments

- rename props to properties in GetSchemaResponse
- Use config for ledger parameters

* Address review comments

* Fix license headers

* deal with lombock stuff causing issues

* Resolve conflict
---
 .../pulsar/broker/admin/v2/SchemasResource.java| 242 +
 .../service/schema/BookkeeperSchemaStorage.java|  29 ++-
 .../service/schema/SchemaRegistryService.java  |   3 +-
 .../broker/service/schema/SchemaStorage.java   |   2 +
 .../pulsar/broker/service/schema/StoredSchema.java |  11 +-
 .../org/apache/pulsar/broker/admin/AdminTest.java  |  28 ++-
 .../java/org/apache/pulsar/client/api/Schema.java  |   9 +
 .../org/apache/pulsar/client/impl/ClientCnx.java   |  49 ++---
 .../apache/pulsar/client/impl/ConsumerImpl.java|   2 +-
 .../apache/pulsar/client/impl/ProducerImpl.java|  15 +-
 .../Schema.java => impl/ProducerResponse.java} |  24 +-
 .../org/apache/pulsar/common/api/Commands.java |  51 -
 .../apache/pulsar/common/api/proto/PulsarApi.java  | 112 ++
 .../pulsar/common/schema/DeleteSchemaResponse.java |  22 +-
 .../pulsar/common/schema/GetSchemaResponse.java|  27 +--
 .../pulsar/common/schema/PostSchemaPayload.java|  23 +-
 .../pulsar/common/schema/PostSchemaResponse.java   |  22 +-
 .../apache/pulsar/common/schema/SchemaInfo.java|  24 +-
 pulsar-common/src/main/proto/PulsarApi.proto   |   2 +-
 19 files changed, 457 insertions(+), 240 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/SchemasResource.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/SchemasResource.java
new file mode 100644
index 000..2173617
--- /dev/null
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/SchemasResource.java
@@ -0,0 +1,242 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under 

[GitHub] merlimat commented on issue #1516: Added test to ensure property name can contain '_' characters.

[GitBox]

merlimat commented on issue #1516: Added test to ensure property name can 
contain '_' characters.
URL: https://github.com/apache/incubator-pulsar/pull/1516#issuecomment-379402597
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1516: Added test to ensure property name can contain '_' characters.

[GitBox]

merlimat commented on issue #1516: Added test to ensure property name can 
contain '_' characters.
URL: https://github.com/apache/incubator-pulsar/pull/1516#issuecomment-379402597
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mgodave commented on issue #1505: Proxy admin web requests

[GitBox]

mgodave commented on issue #1505: Proxy admin web requests
URL: https://github.com/apache/incubator-pulsar/pull/1505#issuecomment-379398871
 
 
   That's fine. I still need to test something so don't merge either way.
   
   On Fri, Apr 6, 2018 at 4:10 PM, Matteo Merli 
   wrote:
   
   > testOutOfOrderDeletePersistenceIntoLedgerWithClose failed
   >
   > This PR was branched when we had the master broken
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1505: Proxy admin web requests

[GitBox]

merlimat commented on issue #1505: Proxy admin web requests
URL: https://github.com/apache/incubator-pulsar/pull/1505#issuecomment-379398302
 
 
   > testOutOfOrderDeletePersistenceIntoLedgerWithClose failed
   
   This PR was branched when we had the master broken


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1381: Schema registry 4/N

[GitBox]

sijie commented on issue #1381: Schema registry 4/N
URL: https://github.com/apache/incubator-pulsar/pull/1381#issuecomment-379391812
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Readme for integration tests (#1518)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new cc28dd0  Readme for integration tests (#1518)
cc28dd0 is described below

commit cc28dd027f02249233e37b73ca08e5e49a8069ae
Author: Ivan Kelly 
AuthorDate: Fri Apr 6 23:36:32 2018 +0200

Readme for integration tests (#1518)
---
 tests/README.md | 22 ++
 1 file changed, 22 insertions(+)

diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 000..2fb4e8f
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,22 @@
+This directory contains integration tests for Pulsar.
+
+The integration tests use a framework called [Arquillian 
Cube](https://github.com/arquillian/arquillian-cube) to bring up a bunch of 
docker containers running Pulsar services. TestNG can then be used to test 
functionallity against these containers.
+
+Arquillian sets up a clean set of containers for each test suite. However, if 
maven is configured to fork per test class, Arquillian will set up a clean set 
of containers _per test class_. The test cases within the class will share the 
environment, so tests which leave the cluster in a different state after 
running should be put in their own test class.
+
+The tests require that docker is installed and running. Tests will only run if 
the integrationTests system property is defined. To run the tests:
+```shell
+# in the top level directory
+pulsar/ $ mvn install -DskipTests -Pdocker # builds the docker images
+...
+pulsar/ $ mvn -f tests/pom.xml test -DintegrationTests
+```
+
+The directories are as follows:
+
+- docker-images/ : Docker images for integration testing.
+- integration/ : The integration tests themselves.
+- integration-tests-base/ : A base module for integration test modules. 
Contains common settings and dependencies.
+- integration-tests-topologies/ : Arquillian cluster definitions for use in 
integration tests.
+- integration-tests-utils/ : Utilities for working with arquillian test 
clusters.
+

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[GitHub] merlimat closed pull request #1514: Compaction allows keyless messages to pass through

[GitBox]

merlimat closed pull request #1514: Compaction allows keyless messages to pass 
through
URL: https://github.com/apache/incubator-pulsar/pull/1514
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java
index fa629827af..4e628bc5f8 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java
@@ -105,12 +105,15 @@ public static boolean isBatch(RawMessage msg) {
 ByteBuf singleMessagePayload = 
Commands.deSerializeSingleMessageInBatch(payload,

 singleMessageMetadataBuilder,

 0, batchSize);
-String key = singleMessageMetadataBuilder.getPartitionKey();
 MessageId id = new 
BatchMessageIdImpl(msg.getMessageIdData().getLedgerId(),
   
msg.getMessageIdData().getEntryId(),
   
msg.getMessageIdData().getPartition(),
   i);
-if (filter.test(key, id)) {
+if (!singleMessageMetadataBuilder.hasPartitionKey()) {
+messagesRetained++;
+
Commands.serializeSingleMessageInBatchWithPayload(singleMessageMetadataBuilder,
+  
singleMessagePayload, batchBuffer);
+} else if 
(filter.test(singleMessageMetadataBuilder.getPartitionKey(), id)) {
 messagesRetained++;
 
Commands.serializeSingleMessageInBatchWithPayload(singleMessageMetadataBuilder,
   
singleMessagePayload, batchBuffer);
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java
index f12870b7ba..fbad47ea30 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java
@@ -123,7 +123,9 @@ private void phaseOneLoop(RawReader reader,
 }
 } else {
 String key = extractKey(m);
-latestForKey.put(key, id);
+if (key != null) {
+latestForKey.put(key, id);
+}
 }
 
 if (id.compareTo(lastMessageId) == 0) {
@@ -213,7 +215,9 @@ private void phaseTwoLoop(RawReader reader, MessageId to, 
Map
 }
 } else {
 String key = extractKey(m);
-if (latestForKey.get(key).equals(id)) {
+if (key == null) { // pass through messages without a 
key
+messageToAdd = Optional.of(m);
+} else if (latestForKey.get(key).equals(id)) {
 messageToAdd = Optional.of(m);
 } else {
 m.close();
@@ -306,7 +310,11 @@ private void phaseTwoLoop(RawReader reader, MessageId to, 
Map
 private static String extractKey(RawMessage m) {
 ByteBuf headersAndPayload = m.getHeadersAndPayload();
 MessageMetadata msgMetadata = 
Commands.parseMessageMetadata(headersAndPayload);
-return msgMetadata.getPartitionKey();
+if (msgMetadata.hasPartitionKey()) {
+return msgMetadata.getPartitionKey();
+} else {
+return null;
+}
 }
 
 private static class PhaseOneResult {
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java 
b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
index af39e12de3..a0f0f972e4 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
@@ -442,4 +442,74 @@ public void testWholeBatchCompactedOut() throws Exception {
 Assert.assertEquals(new String(message.getData()), "my-message-4");
 }
 }
+
+@Test
+public void 

[GitHub] merlimat closed pull request #1518: Readme for integration tests

[GitBox]

merlimat closed pull request #1518: Readme for integration tests
URL: https://github.com/apache/incubator-pulsar/pull/1518
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 00..2fb4e8f58d
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,22 @@
+This directory contains integration tests for Pulsar.
+
+The integration tests use a framework called [Arquillian 
Cube](https://github.com/arquillian/arquillian-cube) to bring up a bunch of 
docker containers running Pulsar services. TestNG can then be used to test 
functionallity against these containers.
+
+Arquillian sets up a clean set of containers for each test suite. However, if 
maven is configured to fork per test class, Arquillian will set up a clean set 
of containers _per test class_. The test cases within the class will share the 
environment, so tests which leave the cluster in a different state after 
running should be put in their own test class.
+
+The tests require that docker is installed and running. Tests will only run if 
the integrationTests system property is defined. To run the tests:
+```shell
+# in the top level directory
+pulsar/ $ mvn install -DskipTests -Pdocker # builds the docker images
+...
+pulsar/ $ mvn -f tests/pom.xml test -DintegrationTests
+```
+
+The directories are as follows:
+
+- docker-images/ : Docker images for integration testing.
+- integration/ : The integration tests themselves.
+- integration-tests-base/ : A base module for integration test modules. 
Contains common settings and dependencies.
+- integration-tests-topologies/ : Arquillian cluster definitions for use in 
integration tests.
+- integration-tests-utils/ : Utilities for working with arquillian test 
clusters.
+


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Compaction allows keyless messages to pass through (#1514)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 346f831  Compaction allows keyless messages to pass through (#1514)
346f831 is described below

commit 346f8312484cc4389d96f2aeefc0be6393c4fcc2
Author: Ivan Kelly 
AuthorDate: Fri Apr 6 23:35:56 2018 +0200

Compaction allows keyless messages to pass through (#1514)

If a message has no key it's impossible to know if a later message
supercedes it, so in this case, it should be simply passed through
compaction (i.e. it should appear when reading from a compacted
topic).
---
 .../pulsar/client/impl/RawBatchConverter.java  |  7 ++-
 .../pulsar/compaction/TwoPhaseCompactor.java   | 14 -
 .../apache/pulsar/compaction/CompactionTest.java   | 70 ++
 3 files changed, 86 insertions(+), 5 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java
index fa62982..4e628bc 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java
@@ -105,12 +105,15 @@ public class RawBatchConverter {
 ByteBuf singleMessagePayload = 
Commands.deSerializeSingleMessageInBatch(payload,

 singleMessageMetadataBuilder,

 0, batchSize);
-String key = singleMessageMetadataBuilder.getPartitionKey();
 MessageId id = new 
BatchMessageIdImpl(msg.getMessageIdData().getLedgerId(),
   
msg.getMessageIdData().getEntryId(),
   
msg.getMessageIdData().getPartition(),
   i);
-if (filter.test(key, id)) {
+if (!singleMessageMetadataBuilder.hasPartitionKey()) {
+messagesRetained++;
+
Commands.serializeSingleMessageInBatchWithPayload(singleMessageMetadataBuilder,
+  
singleMessagePayload, batchBuffer);
+} else if 
(filter.test(singleMessageMetadataBuilder.getPartitionKey(), id)) {
 messagesRetained++;
 
Commands.serializeSingleMessageInBatchWithPayload(singleMessageMetadataBuilder,
   
singleMessagePayload, batchBuffer);
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java
index f12870b..fbad47e 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java
@@ -123,7 +123,9 @@ public class TwoPhaseCompactor extends Compactor {
 }
 } else {
 String key = extractKey(m);
-latestForKey.put(key, id);
+if (key != null) {
+latestForKey.put(key, id);
+}
 }
 
 if (id.compareTo(lastMessageId) == 0) {
@@ -213,7 +215,9 @@ public class TwoPhaseCompactor extends Compactor {
 }
 } else {
 String key = extractKey(m);
-if (latestForKey.get(key).equals(id)) {
+if (key == null) { // pass through messages without a 
key
+messageToAdd = Optional.of(m);
+} else if (latestForKey.get(key).equals(id)) {
 messageToAdd = Optional.of(m);
 } else {
 m.close();
@@ -306,7 +310,11 @@ public class TwoPhaseCompactor extends Compactor {
 private static String extractKey(RawMessage m) {
 ByteBuf headersAndPayload = m.getHeadersAndPayload();
 MessageMetadata msgMetadata = 
Commands.parseMessageMetadata(headersAndPayload);
-return msgMetadata.getPartitionKey();
+if (msgMetadata.hasPartitionKey()) {
+return msgMetadata.getPartitionKey();
+} else {
+return null;
+}
 }
 
 private static class PhaseOneResult {
diff --git 

[GitHub] merlimat opened a new pull request #1519: Avoid contention in ManagedCursorImpl generated by locking on pendingMarkDeleteOps

[GitBox]

merlimat opened a new pull request #1519: Avoid contention in ManagedCursorImpl 
generated by locking on pendingMarkDeleteOps
URL: https://github.com/apache/incubator-pulsar/pull/1519
 
 
   ### Motivation
   
   I have spotted contention in broker threads, that ultimately leads to bumps 
in the latencies.
   This happens when running at high-rates with very little batching.
   
   The synchronizing on `pendingMarkDeleteOps` was not really necessary to 
ensure the correctness.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1501: Rest endpoint to query compaction status

[GitBox]

merlimat closed pull request #1501: Rest endpoint to query compaction status
URL: https://github.com/apache/incubator-pulsar/pull/1501
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
index 86e68583d3..9d82a2fdef 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
@@ -74,6 +74,7 @@
 import org.apache.pulsar.common.api.proto.PulsarApi.KeyValue;
 import org.apache.pulsar.common.api.proto.PulsarApi.MessageMetadata;
 import 
org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.InitialPosition;
+import org.apache.pulsar.common.compaction.CompactionStatus;
 import org.apache.pulsar.common.compression.CompressionCodec;
 import org.apache.pulsar.common.compression.CompressionCodecProvider;
 import org.apache.pulsar.common.naming.TopicDomain;
@@ -1086,6 +1087,12 @@ protected void internalTriggerCompaction(boolean 
authoritative) {
 }
 }
 
+protected CompactionStatus internalCompactionStatus(boolean authoritative) 
{
+validateAdminOperationOnTopic(authoritative);
+PersistentTopic topic = (PersistentTopic) getTopicReference(topicName);
+return topic.compactionStatus();
+}
+
 public static CompletableFuture 
getPartitionedTopicMetadata(PulsarService pulsar,
 String clientAppId, AuthenticationDataSource authenticationData, 
TopicName topicName) {
 CompletableFuture metadataFuture = new 
CompletableFuture<>();
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
index f28d229e0f..b03b533524 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
@@ -40,6 +40,7 @@
 import org.apache.pulsar.broker.admin.impl.PersistentTopicsBase;
 import org.apache.pulsar.client.api.MessageId;
 import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.compaction.CompactionStatus;
 import org.apache.pulsar.common.partition.PartitionedTopicMetadata;
 import org.apache.pulsar.common.policies.data.AuthAction;
 import org.apache.pulsar.common.policies.data.PartitionedTopicStats;
@@ -435,4 +436,17 @@ public void compact(@PathParam("property") String 
property, @PathParam("cluster"
 internalTriggerCompaction(authoritative);
 }
 
+@GET
+@Path("/{property}/{cluster}/{namespace}/{topic}/compaction")
+@ApiOperation(value = "Get the status of a compaction operation for a 
topic.")
+@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have 
admin permission"),
+@ApiResponse(code = 405, message = "Operation not 
allowed on persistent topic"),
+@ApiResponse(code = 404, message = "Topic does not 
exist, or compaction hasn't run") })
+public CompactionStatus compactionStatus(
+@PathParam("property") String property, @PathParam("cluster") 
String cluster,
+@PathParam("namespace") String namespace, @PathParam("topic") 
@Encoded String encodedTopic,
+@QueryParam("authoritative") @DefaultValue("false") boolean 
authoritative) {
+validateTopicName(property, cluster, namespace, encodedTopic);
+return internalCompactionStatus(authoritative);
+}
 }
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java
index 3ffdde8721..4b887434db 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java
@@ -40,6 +40,7 @@
 import org.apache.pulsar.broker.admin.impl.PersistentTopicsBase;
 import org.apache.pulsar.client.api.MessageId;
 import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.compaction.CompactionStatus;
 import org.apache.pulsar.common.partition.PartitionedTopicMetadata;
 import org.apache.pulsar.common.policies.data.AuthAction;
 import org.apache.pulsar.common.policies.data.PartitionedTopicStats;
@@ -407,4 +408,18 @@ public void compact(@PathParam("property") String property,
 validateTopicName(property, namespace, encodedTopic);
 internalTriggerCompaction(authoritative);
 }
+

[incubator-pulsar] branch master updated: Rest endpoint to query compaction status (#1501)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 6f36aee  Rest endpoint to query compaction status (#1501)
6f36aee is described below

commit 6f36aeea72178014d5aed980f38585726ca588d7
Author: Ivan Kelly 
AuthorDate: Fri Apr 6 23:12:17 2018 +0200

Rest endpoint to query compaction status (#1501)

Currently returns whether compaction has not run, is running, is
complete or has failed. If it has failed, the last error is returned
also.
---
 .../broker/admin/impl/PersistentTopicsBase.java|  7 +++
 .../pulsar/broker/admin/v1/PersistentTopics.java   | 14 ++
 .../pulsar/broker/admin/v2/PersistentTopics.java   | 15 +++
 .../broker/service/persistent/PersistentTopic.java | 24 ++
 .../apache/pulsar/broker/admin/AdminApiTest.java   | 36 +++
 .../pulsar/client/admin/PersistentTopics.java  |  8 
 .../admin/internal/PersistentTopicsImpl.java   | 13 ++
 .../pulsar/common/compaction/CompactionStatus.java | 52 ++
 8 files changed, 169 insertions(+)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
index aa52c64..03a435d 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
@@ -74,6 +74,7 @@ import org.apache.pulsar.common.api.Commands;
 import org.apache.pulsar.common.api.proto.PulsarApi.KeyValue;
 import org.apache.pulsar.common.api.proto.PulsarApi.MessageMetadata;
 import 
org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.InitialPosition;
+import org.apache.pulsar.common.compaction.CompactionStatus;
 import org.apache.pulsar.common.compression.CompressionCodec;
 import org.apache.pulsar.common.compression.CompressionCodecProvider;
 import org.apache.pulsar.common.naming.TopicDomain;
@@ -1086,6 +1087,12 @@ public class PersistentTopicsBase extends AdminResource {
 }
 }
 
+protected CompactionStatus internalCompactionStatus(boolean authoritative) 
{
+validateAdminOperationOnTopic(authoritative);
+PersistentTopic topic = (PersistentTopic) getTopicReference(topicName);
+return topic.compactionStatus();
+}
+
 public static CompletableFuture 
getPartitionedTopicMetadata(PulsarService pulsar,
 String clientAppId, AuthenticationDataSource authenticationData, 
TopicName topicName) {
 CompletableFuture metadataFuture = new 
CompletableFuture<>();
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
index f28d229..b03b533 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
@@ -40,6 +40,7 @@ import javax.ws.rs.core.Response;
 import org.apache.pulsar.broker.admin.impl.PersistentTopicsBase;
 import org.apache.pulsar.client.api.MessageId;
 import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.compaction.CompactionStatus;
 import org.apache.pulsar.common.partition.PartitionedTopicMetadata;
 import org.apache.pulsar.common.policies.data.AuthAction;
 import org.apache.pulsar.common.policies.data.PartitionedTopicStats;
@@ -435,4 +436,17 @@ public class PersistentTopics extends PersistentTopicsBase 
{
 internalTriggerCompaction(authoritative);
 }
 
+@GET
+@Path("/{property}/{cluster}/{namespace}/{topic}/compaction")
+@ApiOperation(value = "Get the status of a compaction operation for a 
topic.")
+@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have 
admin permission"),
+@ApiResponse(code = 405, message = "Operation not 
allowed on persistent topic"),
+@ApiResponse(code = 404, message = "Topic does not 
exist, or compaction hasn't run") })
+public CompactionStatus compactionStatus(
+@PathParam("property") String property, @PathParam("cluster") 
String cluster,
+@PathParam("namespace") String namespace, @PathParam("topic") 
@Encoded String encodedTopic,
+@QueryParam("authoritative") @DefaultValue("false") boolean 
authoritative) {
+validateTopicName(property, cluster, namespace, encodedTopic);
+return internalCompactionStatus(authoritative);
+}
 }
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java
 

[incubator-pulsar] branch master updated: Also log bookkeeper (and managed ledger) in tests (#1512)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new c2fb795  Also log bookkeeper (and managed ledger) in tests (#1512)
c2fb795 is described below

commit c2fb7958391f60702d00ef5bd2bc7c982b9cb188
Author: Ivan Kelly 
AuthorDate: Fri Apr 6 23:11:37 2018 +0200

Also log bookkeeper (and managed ledger) in tests (#1512)

Root level is warn, so we need to explicitly set org.apache.bookkeeper
to info to see the logs.
---
 buildtools/src/main/resources/log4j2.xml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/buildtools/src/main/resources/log4j2.xml 
b/buildtools/src/main/resources/log4j2.xml
index e732953..85a7c1e 100644
--- a/buildtools/src/main/resources/log4j2.xml
+++ b/buildtools/src/main/resources/log4j2.xml
@@ -30,5 +30,6 @@
 
 
 
+
 
 

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[GitHub] merlimat closed pull request #1512: Also log bookkeeper (and managed ledger) in tests

[GitBox]

merlimat closed pull request #1512: Also log bookkeeper (and managed ledger) in 
tests
URL: https://github.com/apache/incubator-pulsar/pull/1512
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/buildtools/src/main/resources/log4j2.xml 
b/buildtools/src/main/resources/log4j2.xml
index e7329532ca..85a7c1e59c 100644
--- a/buildtools/src/main/resources/log4j2.xml
+++ b/buildtools/src/main/resources/log4j2.xml
@@ -30,5 +30,6 @@
 
 
 
+
 
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #1518: Readme for integration tests

[GitBox]

ivankelly opened a new pull request #1518: Readme for integration tests
URL: https://github.com/apache/incubator-pulsar/pull/1518
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1504: Enable FQFN specification for the Pulsar Functions CLI

[GitBox]

merlimat closed pull request #1504: Enable FQFN specification for the Pulsar 
Functions CLI
URL: https://github.com/apache/incubator-pulsar/pull/1504
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-client-tools-test/src/test/java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java
 
b/pulsar-client-tools-test/src/test/java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java
index 7bcb1090c5..5f745d15a8 100644
--- 
a/pulsar-client-tools-test/src/test/java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java
+++ 
b/pulsar-client-tools-test/src/test/java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java
@@ -232,6 +232,31 @@ public void testCreateWithoutNamespace() throws Exception {
 verify(functions, times(1)).createFunction(any(FunctionConfig.class), 
anyString());
 }
 
+@Test
+public void testCreateUsingFullyQualifiedFunctionName() throws Exception {
+String inputTopicName = TEST_NAME + "-input-topic";
+String outputTopicName = TEST_NAME + "-output-topic";
+String tenant = "sample";
+String namespace = "ns1";
+String functionName = "func";
+String fqfn = String.format("%s/%s/%s", tenant, namespace, 
functionName);
+
+cmd.run(new String[] {
+"create",
+"--inputs", inputTopicName,
+"--output", outputTopicName,
+"--fqfn", fqfn,
+"--jar", "SomeJar.jar",
+"--className", DummyFunction.class.getName(),
+});
+
+CreateFunction creater = cmd.getCreater();
+assertEquals(tenant, creater.getFunctionConfig().getTenant());
+assertEquals(namespace, creater.getFunctionConfig().getNamespace());
+assertEquals(functionName, creater.getFunctionConfig().getName());
+verify(functions, times(1)).createFunction(any(FunctionConfig.class), 
anyString());
+}
+
 @Test
 public void testCreateWithoutFunctionName() throws Exception {
 String inputTopicName = TEST_NAME + "-input-topic";
diff --git 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
index e3951abdcb..adf7283b13 100644
--- 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
+++ 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
@@ -103,6 +103,9 @@ void processArguments() throws Exception {}
  */
 @Getter
 abstract class NamespaceCommand extends BaseCommand {
+@Parameter(names = "--fqfn", description = "The Fully Qualified 
Function Name (FQFN) for the function", required = false)
+protected String fqfn;
+
 @Parameter(names = "--tenant", description = "The function's tenant", 
required = true)
 protected String tenant;
 
@@ -124,6 +127,8 @@ void processArguments() throws Exception {}
  */
 @Getter
 abstract class FunctionConfigCommand extends BaseCommand {
+@Parameter(names = "--fqfn", description = "The Fully Qualified 
Function Name (FQFN) for the function")
+protected String fqfn;
 @Parameter(names = "--tenant", description = "The function's tenant")
 protected String tenant;
 @Parameter(names = "--namespace", description = "The function's 
namespace")
@@ -168,13 +173,29 @@ void processArguments() throws Exception {}
 
 @Override
 void processArguments() throws Exception {
-
 FunctionConfig.Builder functionConfigBuilder;
+
+// Initialize config builder either from a supplied YAML config 
file or from scratch
 if (null != fnConfigFile) {
 functionConfigBuilder = loadConfig(new File(fnConfigFile));
 } else {
 functionConfigBuilder = FunctionConfig.newBuilder();
 }
+
+if (null != fqfn) {
+parseFullyQualifiedFunctionName(fqfn, functionConfigBuilder);
+} else {
+if (null != tenant) {
+functionConfigBuilder.setTenant(tenant);
+}
+if (null != namespace) {
+functionConfigBuilder.setNamespace(namespace);
+}
+if (null != functionName) {
+functionConfigBuilder.setName(functionName);
+}
+}
+
 if (null != inputs) {
 String[] topicNames = inputs.split(",");
 for (int i = 0; i < topicNames.length; ++i) {
@@ -192,15 +213,6 @@ void processArguments() throws Exception {
 if (null != logTopic) {
 functionConfigBuilder.setLogTopic(logTopic);
 }
-   

[incubator-pulsar] branch master updated: Enable FQFN specification for the Pulsar Functions CLI (#1504)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 9d23892  Enable FQFN specification for the Pulsar Functions CLI (#1504)
9d23892 is described below

commit 9d23892568e4158c04417bab027dae7fc8b6eebc
Author: Luc Perkins 
AuthorDate: Fri Apr 6 12:34:47 2018 -0700

Enable FQFN specification for the Pulsar Functions CLI (#1504)

* Add FQFN flag plus corresponding test

* add fqfn field to function config protobuf definition

* add section in PF overview on FQFNs

* add missing --jar flag to test

* remove unnecessary imports

* add missing license header

* add final missing import

* remove incomplete line

* make FQFN name check more stringent
---
 .../apache/pulsar/admin/cli/CmdFunctionsTest.java  | 25 
 .../org/apache/pulsar/admin/cli/CmdFunctions.java  | 45 --
 .../proto/src/main/proto/Function.proto|  3 ++
 site/_includes/fqfn.html   | 23 +++
 site/_sass/_docs.scss  |  6 +--
 site/docs/latest/functions/overview.md |  8 
 6 files changed, 96 insertions(+), 14 deletions(-)

diff --git 
a/pulsar-client-tools-test/src/test/java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java
 
b/pulsar-client-tools-test/src/test/java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java
index 7bcb109..5f745d1 100644
--- 
a/pulsar-client-tools-test/src/test/java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java
+++ 
b/pulsar-client-tools-test/src/test/java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java
@@ -233,6 +233,31 @@ public class CmdFunctionsTest {
 }
 
 @Test
+public void testCreateUsingFullyQualifiedFunctionName() throws Exception {
+String inputTopicName = TEST_NAME + "-input-topic";
+String outputTopicName = TEST_NAME + "-output-topic";
+String tenant = "sample";
+String namespace = "ns1";
+String functionName = "func";
+String fqfn = String.format("%s/%s/%s", tenant, namespace, 
functionName);
+
+cmd.run(new String[] {
+"create",
+"--inputs", inputTopicName,
+"--output", outputTopicName,
+"--fqfn", fqfn,
+"--jar", "SomeJar.jar",
+"--className", DummyFunction.class.getName(),
+});
+
+CreateFunction creater = cmd.getCreater();
+assertEquals(tenant, creater.getFunctionConfig().getTenant());
+assertEquals(namespace, creater.getFunctionConfig().getNamespace());
+assertEquals(functionName, creater.getFunctionConfig().getName());
+verify(functions, times(1)).createFunction(any(FunctionConfig.class), 
anyString());
+}
+
+@Test
 public void testCreateWithoutFunctionName() throws Exception {
 String inputTopicName = TEST_NAME + "-input-topic";
 String outputTopicName = TEST_NAME + "-output-topic";
diff --git 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
index e3951ab..adf7283 100644
--- 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
+++ 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
@@ -103,6 +103,9 @@ public class CmdFunctions extends CmdBase {
  */
 @Getter
 abstract class NamespaceCommand extends BaseCommand {
+@Parameter(names = "--fqfn", description = "The Fully Qualified 
Function Name (FQFN) for the function", required = false)
+protected String fqfn;
+
 @Parameter(names = "--tenant", description = "The function's tenant", 
required = true)
 protected String tenant;
 
@@ -124,6 +127,8 @@ public class CmdFunctions extends CmdBase {
  */
 @Getter
 abstract class FunctionConfigCommand extends BaseCommand {
+@Parameter(names = "--fqfn", description = "The Fully Qualified 
Function Name (FQFN) for the function")
+protected String fqfn;
 @Parameter(names = "--tenant", description = "The function's tenant")
 protected String tenant;
 @Parameter(names = "--namespace", description = "The function's 
namespace")
@@ -168,13 +173,29 @@ public class CmdFunctions extends CmdBase {
 
 @Override
 void processArguments() throws Exception {
-
 FunctionConfig.Builder functionConfigBuilder;
+
+// Initialize config builder either from a supplied YAML config 
file or from scratch
 if (null != fnConfigFile) {
 functionConfigBuilder = loadConfig(new File(fnConfigFile));
 } else {
 functionConfigBuilder = 

[GitHub] ivankelly commented on issue #1512: Also log bookkeeper (and managed ledger) in tests

[GitBox]

ivankelly commented on issue #1512: Also log bookkeeper (and managed ledger) in 
tests
URL: https://github.com/apache/incubator-pulsar/pull/1512#issuecomment-379355022
 
 
   retest this please // c++ test timed out "Build timed out (after 200 
minutes). Marking the build as aborted."


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new issue #1517: Reader#hasMessageAvailable can report false when it should be true

[GitBox]

ivankelly opened a new issue #1517: Reader#hasMessageAvailable can report false 
when it should be true
URL: https://github.com/apache/incubator-pulsar/issues/1517
 
 
    Expected behavior
   
   true
   
    Actual behavior
   
   false
   
    Steps to reproduce
   
   1. Write to a topic
   2. Restart the broker
   2. Create a Reader reading from very start of topic
   4. Call hasMessageAvailable, will return false
   
   Test case: 
https://github.com/ivankelly/incubator-pulsar/commit/362ab39eb341e58dede568cc0c7e9321a967adfd
   
   This is happening, because managed ledger getLastPosition() uses 
lastConfirmedEntry, which is set to ```:-1```, when a new new ledger 
is created (as happens on restart). When the client gets this, and the client 
hasn't already received a message, 
   
   
   ```
   public CompletableFuture hasMessageAvailableAsync() {
   final CompletableFuture booleanFuture = new 
CompletableFuture<>();
   
   if (lastMessageIdInBroker.compareTo(lastDequeuedMessage) > 0 &&
   ((MessageIdImpl)lastMessageIdInBroker).getEntryId() != -1) {
   booleanFuture.complete(true);
   } else {
   getLastMessageIdAsync().thenAccept(messageId -> {
   lastMessageIdInBroker = messageId;
   
   if (lastMessageIdInBroker.compareTo(lastDequeuedMessage) > 0 
&&
   ((MessageIdImpl)lastMessageIdInBroker).getEntryId() != 
-1) {
   booleanFuture.complete(true);
   } else {
   booleanFuture.complete(false);
   }
   }).exceptionally(e -> {
   log.error("[{}][{}] Failed getLastMessageId command", topic, 
subscription);
   booleanFuture.completeExceptionally(e.getCause());
   return null;
   });
   }
   return booleanFuture;
   }
   ```
   The method gets ```:-1``` from the broker, which hits the false 
clause every time.
   
   The solution would be to check in managed ledger if nothing has been 
written, and if so, find the last entry in the last non-empty ledger. I wanted 
to check why it wasn't done like this before implementing though. @zhaijack 
@merlimat @sijie 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Message deduplication documentation (#1271)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new ec210cb  Message deduplication documentation (#1271)
ec210cb is described below

commit ec210cb0c0648a64a80d4b30c312c82420588836
Author: Luc Perkins 
AuthorDate: Fri Apr 6 12:04:06 2018 -0700

Message deduplication documentation (#1271)

* add deduplication diagram

* add CLI docs to YAML file

* add docs for new broker.conf settings

* add link to streamlio blog post and remove errant TODOs

* add more to theory section

* add producer idempotency section

* add message deduplication cookbook

* add multiple new sections to cookbook

* add note about dedup and namespaces

* add new badge to new docs

* update description of configs

* finish admin section of doc

* fix minor markdown error

* fix merge conflicts in gemfile.lock
---
 conf/broker.conf   |   6 +-
 .../apache/pulsar/client/api/ProducerBuilder.java  |   2 +
 site/Gemfile.lock  |   2 +-
 site/_data/cli/pulsar-admin.yaml   |  10 ++
 site/_data/config/broker.yaml  |  14 ++-
 site/_data/sidebar.yaml|   6 +-
 site/docs/latest/admin/ZooKeeperBookKeeper.md  |   2 +-
 site/docs/latest/clients/Java.md   |   2 +-
 .../latest/{advanced => cookbooks}/Encryption.md   |   0
 .../{advanced => cookbooks}/PartitionedTopics.md   |   0
 .../{advanced => cookbooks}/RetentionExpiry.md |   0
 .../docs/latest/cookbooks/message-deduplication.md | 121 +
 .../getting-started/ConceptsAndArchitecture.md |  33 +-
 site/img/message-deduplication.png | Bin 0 -> 116703 bytes
 14 files changed, 186 insertions(+), 12 deletions(-)

diff --git a/conf/broker.conf b/conf/broker.conf
index d78fbbc..d38abf7 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -86,9 +86,9 @@ brokerDeduplicationEnabled=false
 brokerDeduplicationMaxNumberOfProducers=1
 
 # Number of entries after which a dedup info snapshot is taken.
-# A bigger interval will lead to less snapshots being taken though it would
-# increase the topic recovery time, when the entries published after the
-# snapshot need to be replayed
+# A larger interval will lead to fewer snapshots being taken, though it would
+# increase the topic recovery time when the entries published after the
+# snapshot need to be replayed.
 brokerDeduplicationEntriesInterval=1000
 
 # Time of inactivity after which the broker will discard the deduplication 
information
diff --git 
a/pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java 
b/pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
index 16dcd00..03d82a7 100644
--- 
a/pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
+++ 
b/pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
@@ -101,6 +101,8 @@ public interface ProducerBuilder extends Serializable, 
Cloneable {
  * Set the send timeout (default: 30 seconds)
  * 
  * If a message is not acknowledged by the server before the sendTimeout 
expires, an error will be reported.
+ * Setting the timeout to zero, for example setTimeout(0, 
TimeUnit.SECONDS) will set the timeout
+ * to infinity, which can be useful when using Pulsar's message 
deduplication feature. 
  *
  * @param sendTimeout
  *the send timeout
diff --git a/site/Gemfile.lock b/site/Gemfile.lock
index 1eb692b..be5c876 100644
--- a/site/Gemfile.lock
+++ b/site/Gemfile.lock
@@ -47,7 +47,7 @@ GEM
 rouge (3.1.1)
 ruby_dep (1.5.0)
 safe_yaml (1.0.4)
-sass (3.5.5)
+sass (3.5.6)
   sass-listen (~> 4.0.0)
 sass-listen (4.0.0)
   rb-fsevent (~> 0.9, >= 0.9.4)
diff --git a/site/_data/cli/pulsar-admin.yaml b/site/_data/cli/pulsar-admin.yaml
index ef56909..424c2b4 100644
--- a/site/_data/cli/pulsar-admin.yaml
+++ b/site/_data/cli/pulsar-admin.yaml
@@ -326,6 +326,16 @@ commands:
   - name: delete
 description: Deletes a namespace
 argument: property/cluster/namespace
+  - name: set-deduplication
+description: Enable or disable message deduplication on a namespace
+argument: property/cluster/namespace
+options:
+- flags: --enable, -e
+  description: Enable message deduplication on the specified namespace
+  default: 'false'
+- flags: --disable, -d
+  description: Disable message deduplication on the specified namespace
+  default: 'false'
   - name: permissions
 description: Get the permissions on a namespace
 argument: property/cluster/namespace
diff --git 

[GitHub] merlimat closed pull request #1271: Message deduplication documentation

[GitBox]

merlimat closed pull request #1271: Message deduplication documentation
URL: https://github.com/apache/incubator-pulsar/pull/1271
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/conf/broker.conf b/conf/broker.conf
index d78fbbc02d..d38abf7046 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -86,9 +86,9 @@ brokerDeduplicationEnabled=false
 brokerDeduplicationMaxNumberOfProducers=1
 
 # Number of entries after which a dedup info snapshot is taken.
-# A bigger interval will lead to less snapshots being taken though it would
-# increase the topic recovery time, when the entries published after the
-# snapshot need to be replayed
+# A larger interval will lead to fewer snapshots being taken, though it would
+# increase the topic recovery time when the entries published after the
+# snapshot need to be replayed.
 brokerDeduplicationEntriesInterval=1000
 
 # Time of inactivity after which the broker will discard the deduplication 
information
diff --git 
a/pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java 
b/pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
index 16dcd009cb..03d82a7858 100644
--- 
a/pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
+++ 
b/pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
@@ -101,6 +101,8 @@
  * Set the send timeout (default: 30 seconds)
  * 
  * If a message is not acknowledged by the server before the sendTimeout 
expires, an error will be reported.
+ * Setting the timeout to zero, for example setTimeout(0, 
TimeUnit.SECONDS) will set the timeout
+ * to infinity, which can be useful when using Pulsar's message 
deduplication feature. 
  *
  * @param sendTimeout
  *the send timeout
diff --git a/site/Gemfile.lock b/site/Gemfile.lock
index 1eb692b4ef..be5c876094 100644
--- a/site/Gemfile.lock
+++ b/site/Gemfile.lock
@@ -47,7 +47,7 @@ GEM
 rouge (3.1.1)
 ruby_dep (1.5.0)
 safe_yaml (1.0.4)
-sass (3.5.5)
+sass (3.5.6)
   sass-listen (~> 4.0.0)
 sass-listen (4.0.0)
   rb-fsevent (~> 0.9, >= 0.9.4)
diff --git a/site/_data/cli/pulsar-admin.yaml b/site/_data/cli/pulsar-admin.yaml
index ef56909cda..424c2b41a2 100644
--- a/site/_data/cli/pulsar-admin.yaml
+++ b/site/_data/cli/pulsar-admin.yaml
@@ -326,6 +326,16 @@ commands:
   - name: delete
 description: Deletes a namespace
 argument: property/cluster/namespace
+  - name: set-deduplication
+description: Enable or disable message deduplication on a namespace
+argument: property/cluster/namespace
+options:
+- flags: --enable, -e
+  description: Enable message deduplication on the specified namespace
+  default: 'false'
+- flags: --disable, -d
+  description: Disable message deduplication on the specified namespace
+  default: 'false'
   - name: permissions
 description: Get the permissions on a namespace
 argument: property/cluster/namespace
diff --git a/site/_data/config/broker.yaml b/site/_data/config/broker.yaml
index 445b4a6d8e..9cc9fb6d9b 100644
--- a/site/_data/config/broker.yaml
+++ b/site/_data/config/broker.yaml
@@ -47,10 +47,22 @@ configs:
   description: Hostname or IP address the service binds on, default is 0.0.0.0.
 - name: advertisedAddress
   default: ''
-  description: Hostname or IP address the service advertises to the outside 
world. If not set, the value of InetAddress.getLocalHost().getHostName() is 
used.
+  description: Hostname or IP address the service advertises to the outside 
world. If not set, the value of `InetAddress.getLocalHost().getHostName()` is 
used.
 - name: clusterName
   default: ''
   description: Name of the cluster to which this broker belongs to
+- name: brokerDeduplicationEnabled
+  default: 'false'
+  description: Sets the default behavior for message deduplication in the 
broker. If enabled, the broker will reject messages that were already stored in 
the topic. This setting can be overridden on a per-namespace basis.
+- name: brokerDeduplicationMaxNumberOfProducers
+  default: '1'
+  description: The maximum number of producers for which information will be 
stored for deduplication purposes.
+- name: brokerDeduplicationEntriesInterval
+  default: '1000'
+  description: The number of entries after which a deduplication informational 
snapshot is taken. A larger interval will lead to fewer snapshots being taken, 
though this would also lengthen the topic recovery time (the time required for 
entries published after the snapshot to be replayed).
+- name: brokerDeduplicationProducerInactivityTimeoutMinutes
+  default: '360'
+  description: The time of inactivity (in minutes) after which the broker will 

[GitHub] merlimat closed pull request #1498: Enable Pulsar Functions to access entire user config map

[GitBox]

merlimat closed pull request #1498: Enable Pulsar Functions to access entire 
user config map
URL: https://github.com/apache/incubator-pulsar/pull/1498
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pulsar-client-cpp/python/functions/context.py 
b/pulsar-client-cpp/python/functions/context.py
index 4e1a969547..c1f30edc94 100644
--- a/pulsar-client-cpp/python/functions/context.py
+++ b/pulsar-client-cpp/python/functions/context.py
@@ -81,7 +81,12 @@ def get_logger(self):
 
   @abstractmethod
   def get_user_config_value(self, key):
-"""Returns the value of the user defined config. If the key doesnt exist 
return null instead"""
+"""Returns the value of the user-defined config. If the key doesn't exist, 
None is returned"""
+pass
+  
+  @abstractmethod
+  def get_user_config_map(self):
+"""Returns the entire user-defined config as a dict (the dict will be 
empty if no user-defined config is supplied)"""
 pass
 
   @abstractmethod
diff --git 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
index a1abf30d8f..c4d78c2ade 100644
--- 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
+++ 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
@@ -21,6 +21,7 @@
 import org.slf4j.Logger;
 
 import java.util.Collection;
+import java.util.Map;
 import java.util.Optional;
 import java.util.concurrent.CompletableFuture;
 
@@ -112,6 +113,12 @@
  */
 void incrCounter(String key, long amount);
 
+/**
+ * Get a map of all user-defined key/value configs for the function
+ * @return The full map of user-defined config values
+ */
+Map getUserConfigMap();
+
 /**
  * Get any user-defined key/value
  * @param key The key
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
index 02fa616a78..3db2010260 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
@@ -184,6 +184,11 @@ public String getUserConfigValueOrDefault(String key, 
String defaultValue) {
 return getUserConfigValue(key).orElse(defaultValue);
 }
 
+@Override
+public Map getUserConfigMap() {
+return config.getFunctionConfig().getUserConfigMap();
+}
+
 @Override
 public void incrCounter(String key, long amount) {
 if (null != stateContext) {
diff --git a/pulsar-functions/instance/src/main/python/contextimpl.py 
b/pulsar-functions/instance/src/main/python/contextimpl.py
index 9c77fd1c39..f950abf964 100644
--- a/pulsar-functions/instance/src/main/python/contextimpl.py
+++ b/pulsar-functions/instance/src/main/python/contextimpl.py
@@ -92,6 +92,9 @@ def get_user_config_value(self, key):
   return str(self.instance_config.function_config.userConfig[key])
 else:
   return None
+  
+  def get_user_config_map(self):
+return self.instance_config.function_config.userConfig
 
   def record_metric(self, metric_name, metric_value):
 if not metric_name in self.accumulated_metrics:


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Enable Pulsar Functions to access entire user config map (#1498)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new b5781a1  Enable Pulsar Functions to access entire user config map 
(#1498)
b5781a1 is described below

commit b5781a15367f234a4c1c2fe23931a4904f2fe893
Author: Luc Perkins 
AuthorDate: Fri Apr 6 11:54:46 2018 -0700

Enable Pulsar Functions to access entire user config map (#1498)

* add full user config map fetch capability to pulsar funtions context

* add full user config map fetch for Python API

* Standardize naming across java and python
---
 pulsar-client-cpp/python/functions/context.py  | 7 ++-
 .../src/main/java/org/apache/pulsar/functions/api/Context.java | 7 +++
 .../java/org/apache/pulsar/functions/instance/ContextImpl.java | 5 +
 pulsar-functions/instance/src/main/python/contextimpl.py   | 3 +++
 4 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/pulsar-client-cpp/python/functions/context.py 
b/pulsar-client-cpp/python/functions/context.py
index 4e1a969..c1f30ed 100644
--- a/pulsar-client-cpp/python/functions/context.py
+++ b/pulsar-client-cpp/python/functions/context.py
@@ -81,7 +81,12 @@ class Context(object):
 
   @abstractmethod
   def get_user_config_value(self, key):
-"""Returns the value of the user defined config. If the key doesnt exist 
return null instead"""
+"""Returns the value of the user-defined config. If the key doesn't exist, 
None is returned"""
+pass
+  
+  @abstractmethod
+  def get_user_config_map(self):
+"""Returns the entire user-defined config as a dict (the dict will be 
empty if no user-defined config is supplied)"""
 pass
 
   @abstractmethod
diff --git 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
index a1abf30..c4d78c2 100644
--- 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
+++ 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.functions.api;
 import org.slf4j.Logger;
 
 import java.util.Collection;
+import java.util.Map;
 import java.util.Optional;
 import java.util.concurrent.CompletableFuture;
 
@@ -113,6 +114,12 @@ public interface Context {
 void incrCounter(String key, long amount);
 
 /**
+ * Get a map of all user-defined key/value configs for the function
+ * @return The full map of user-defined config values
+ */
+Map getUserConfigMap();
+
+/**
  * Get any user-defined key/value
  * @param key The key
  * @return The Optional value specified by the user for that key.
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
index 02fa616..3db2010 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
@@ -185,6 +185,11 @@ class ContextImpl implements Context {
 }
 
 @Override
+public Map getUserConfigMap() {
+return config.getFunctionConfig().getUserConfigMap();
+}
+
+@Override
 public void incrCounter(String key, long amount) {
 if (null != stateContext) {
 stateContext.incr(key, amount);
diff --git a/pulsar-functions/instance/src/main/python/contextimpl.py 
b/pulsar-functions/instance/src/main/python/contextimpl.py
index 9c77fd1..f950abf 100644
--- a/pulsar-functions/instance/src/main/python/contextimpl.py
+++ b/pulsar-functions/instance/src/main/python/contextimpl.py
@@ -92,6 +92,9 @@ class ContextImpl(pulsar.Context):
   return str(self.instance_config.function_config.userConfig[key])
 else:
   return None
+  
+  def get_user_config_map(self):
+return self.instance_config.function_config.userConfig
 
   def record_metric(self, metric_name, metric_value):
 if not metric_name in self.accumulated_metrics:

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[incubator-pulsar] branch master updated: Pulsar Functions diagrams (#1502)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 125ca58  Pulsar Functions diagrams (#1502)
125ca58 is described below

commit 125ca585da6b02ae35c9681ab96aa302f9efdaa0
Author: Luc Perkins 
AuthorDate: Fri Apr 6 11:53:57 2018 -0700

Pulsar Functions diagrams (#1502)

* Add brief section to overview on processing guarantees

* add new sections to overview

* add basic PF diagram

* add word count diagram

* add two new examples to overview

* switch from python to java example

* Update description of counters
---
 site/_sass/_docs.scss |   2 +-
 site/docs/latest/functions/guarantees.md  |   2 +-
 site/docs/latest/functions/overview.md| 139 --
 site/img/pulsar-functions-overview.png| Bin 0 -> 77077 bytes
 site/img/pulsar-functions-routing-example.png | Bin 0 -> 62087 bytes
 site/img/pulsar-functions-word-count.png  | Bin 0 -> 85250 bytes
 6 files changed, 133 insertions(+), 10 deletions(-)

diff --git a/site/_sass/_docs.scss b/site/_sass/_docs.scss
index 42277f0..a66bd16 100644
--- a/site/_sass/_docs.scss
+++ b/site/_sass/_docs.scss
@@ -195,7 +195,7 @@
   border-bottom: 1px solid black;
 
   tr th {
-padding-right: $table-right-padding;
+padding: 0 $table-right-padding .5rem 0;
   }
 }
 
diff --git a/site/docs/latest/functions/guarantees.md 
b/site/docs/latest/functions/guarantees.md
index 541ac55..f6adc31 100644
--- a/site/docs/latest/functions/guarantees.md
+++ b/site/docs/latest/functions/guarantees.md
@@ -10,7 +10,7 @@ Delivery semantics | Description
 :--|:---
 **At-most-once** delivery | Each message that is sent to the function will 
most likely be processed but also may not be (hence the "at most")
 **At-least-once** delivery | Each message that is sent to the function could 
be processed more than once (hence the "at least")
-**Effectively-once** delivery | Each message that is sent to the function will 
have one output associated with it. The function may be invoked more than once, 
perhaps due to some kind of system failure, but the function will produce one 
effect for each incoming message.
+**Effectively-once** delivery | Each message that is sent to the function will 
have one output associated with it
 
 ## Applying processing guarantees to a function
 
diff --git a/site/docs/latest/functions/overview.md 
b/site/docs/latest/functions/overview.md
index 764515d..9629856 100644
--- a/site/docs/latest/functions/overview.md
+++ b/site/docs/latest/functions/overview.md
@@ -10,7 +10,7 @@ preview: true
 * apply a user-supplied processing logic to each message,
 * publish the results of the computation to another topic
 
-Here's an example Pulsar Function for Java:
+Here's an example Pulsar Function for Java (using the [native 
interface](../api#java-native)):
 
 ```java
 import java.util.Function;
@@ -27,9 +27,9 @@ Functions are executed each time a message is published to 
the input topic. If a
 
 The core goal behind Pulsar Functions is to enable you to easily create 
processing logic of any level of complexity without needing to deploy a 
separate neighboring system (such as [Apache Storm](http://storm.apache.org/), 
[Apache Heron](https://apache.github.io/incubator-heron), [Apache 
Flink](https://flink.apache.org/), etc.). Pulsar Functions is essentially 
ready-made compute infrastructure at your disposal as part of your Pulsar 
messaging system. This core goal is tied to a series of [...]
 
-* Developer productive ([language-native](#native) vs. [Pulsar Functions 
SDK](#sdk) functions)
-* easy troubleshooting
-* Operational simplicity (no need for an external system)
+* Developer productivity ([language-native](#native) vs. [Pulsar Functions 
SDK](#sdk) functions)
+* Easy troubleshooting
+* Operational simplicity (no need for an external processing system)
 
 ## Inspirations
 
@@ -41,7 +41,98 @@ The Pulsar Functions feature was inspired by (and takes cues 
from) several syste
 Pulsar Functions could be described as
 
 * [Lambda](https://aws.amazon.com/lambda/)-style functions that are
-* specifically designed to work with Pulsar
+* specifically designed to use Pulsar as a message bus
+
+## Programming model
+
+The core programming model behind Pulsar Functions is very simple:
+
+* Functions receive messages from one or more **input {% popover topics %}**. 
Every time a message is received, the function can do a variety of things:
+  * Apply some processing logic to the input and write output to:
+* An **output topic** in Pulsar
+* [Apache BookKeeper](#state-storage)
+  * Write logs to a **log topic** (potentially for debugging purposes)
+  * Increment a 

[GitHub] merlimat closed pull request #1502: Pulsar Functions diagrams

[GitBox]

merlimat closed pull request #1502: Pulsar Functions diagrams
URL: https://github.com/apache/incubator-pulsar/pull/1502
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/site/_sass/_docs.scss b/site/_sass/_docs.scss
index 42277f070c..a66bd1640a 100644
--- a/site/_sass/_docs.scss
+++ b/site/_sass/_docs.scss
@@ -195,7 +195,7 @@
   border-bottom: 1px solid black;
 
   tr th {
-padding-right: $table-right-padding;
+padding: 0 $table-right-padding .5rem 0;
   }
 }
 
diff --git a/site/docs/latest/functions/guarantees.md 
b/site/docs/latest/functions/guarantees.md
index 541ac55917..f6adc3150d 100644
--- a/site/docs/latest/functions/guarantees.md
+++ b/site/docs/latest/functions/guarantees.md
@@ -10,7 +10,7 @@ Delivery semantics | Description
 :--|:---
 **At-most-once** delivery | Each message that is sent to the function will 
most likely be processed but also may not be (hence the "at most")
 **At-least-once** delivery | Each message that is sent to the function could 
be processed more than once (hence the "at least")
-**Effectively-once** delivery | Each message that is sent to the function will 
have one output associated with it. The function may be invoked more than once, 
perhaps due to some kind of system failure, but the function will produce one 
effect for each incoming message.
+**Effectively-once** delivery | Each message that is sent to the function will 
have one output associated with it
 
 ## Applying processing guarantees to a function
 
diff --git a/site/docs/latest/functions/overview.md 
b/site/docs/latest/functions/overview.md
index 764515d912..9629856886 100644
--- a/site/docs/latest/functions/overview.md
+++ b/site/docs/latest/functions/overview.md
@@ -10,7 +10,7 @@ preview: true
 * apply a user-supplied processing logic to each message,
 * publish the results of the computation to another topic
 
-Here's an example Pulsar Function for Java:
+Here's an example Pulsar Function for Java (using the [native 
interface](../api#java-native)):
 
 ```java
 import java.util.Function;
@@ -27,9 +27,9 @@ Functions are executed each time a message is published to 
the input topic. If a
 
 The core goal behind Pulsar Functions is to enable you to easily create 
processing logic of any level of complexity without needing to deploy a 
separate neighboring system (such as [Apache Storm](http://storm.apache.org/), 
[Apache Heron](https://apache.github.io/incubator-heron), [Apache 
Flink](https://flink.apache.org/), etc.). Pulsar Functions is essentially 
ready-made compute infrastructure at your disposal as part of your Pulsar 
messaging system. This core goal is tied to a series of other goals:
 
-* Developer productive ([language-native](#native) vs. [Pulsar Functions 
SDK](#sdk) functions)
-* easy troubleshooting
-* Operational simplicity (no need for an external system)
+* Developer productivity ([language-native](#native) vs. [Pulsar Functions 
SDK](#sdk) functions)
+* Easy troubleshooting
+* Operational simplicity (no need for an external processing system)
 
 ## Inspirations
 
@@ -41,7 +41,98 @@ The Pulsar Functions feature was inspired by (and takes cues 
from) several syste
 Pulsar Functions could be described as
 
 * [Lambda](https://aws.amazon.com/lambda/)-style functions that are
-* specifically designed to work with Pulsar
+* specifically designed to use Pulsar as a message bus
+
+## Programming model
+
+The core programming model behind Pulsar Functions is very simple:
+
+* Functions receive messages from one or more **input {% popover topics %}**. 
Every time a message is received, the function can do a variety of things:
+  * Apply some processing logic to the input and write output to:
+* An **output topic** in Pulsar
+* [Apache BookKeeper](#state-storage)
+  * Write logs to a **log topic** (potentially for debugging purposes)
+  * Increment a [counter](#counters)
+
+![Pulsar Functions core programming model](/img/pulsar-functions-overview.png)
+
+### Word count example {#word-count}
+
+If you were to implement the classic word count example using Pulsar 
Functions, it might look something like this:
+
+![Pulsar Functions word count example](/img/pulsar-functions-word-count.png)
+
+Here, sentences are produced on the `sentences` topic. The Pulsar Function 
listens on that topic and whenever a message arrives it splits the sentence up 
into individual words and increments a [counter](#counters) for each word every 
time that word is encountered. The value of that counter is then available to 
all [instances](#parallelism) of the function.
+
+If you were writing the function in [Java](../api#java) using the [Pulsar 
Functions SDK for Java](../api#java-sdk), you could write 

[GitHub] lucperkins commented on issue #1502: Pulsar Functions diagrams

[GitBox]

lucperkins commented on issue #1502: Pulsar Functions diagrams
URL: https://github.com/apache/incubator-pulsar/pull/1502#issuecomment-379338645
 
 
   @srkukarni We're unblocked on CI now, so this is ready to merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Consolidate Pair implementations (#1515)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 6e8b5bf  Consolidate Pair implementations (#1515)
6e8b5bf is described below

commit 6e8b5bf05ce0eb31795357879a1ee62297467480
Author: Dave Rusek 
AuthorDate: Fri Apr 6 12:09:14 2018 -0600

Consolidate Pair implementations (#1515)
---
 .../apache/bookkeeper/mledger/impl/EntryCache.java |  2 +-
 .../impl/EntryCacheDefaultEvictionPolicy.java  |  6 +--
 .../bookkeeper/mledger/impl/EntryCacheImpl.java| 14 ++---
 .../bookkeeper/mledger/impl/EntryCacheManager.java |  4 +-
 .../mledger/impl/ManagedCursorContainer.java   |  4 +-
 .../bookkeeper/mledger/impl/ManagedCursorImpl.java |  8 +--
 .../bookkeeper/mledger/impl/ManagedLedgerImpl.java | 16 +++---
 .../mledger/impl/NonDurableCursorImpl.java |  8 +--
 .../org/apache/bookkeeper/mledger/util/Pair.java   | 59 --
 .../apache/bookkeeper/mledger/util/RangeCache.java |  5 +-
 .../bookkeeper/mledger/impl/ManagedLedgerTest.java | 16 +++---
 .../apache/bookkeeper/mledger/util/PairTest.java   | 50 --
 .../bookkeeper/mledger/util/RangeCacheTest.java| 21 
 .../java/org/apache/zookeeper/MockZooKeeper.java   | 40 +++
 .../broker/loadbalance/impl/OverloadShedder.java   | 12 ++---
 .../impl/AcknowledgmentsGroupingTracker.java   |  3 +-
 pulsar-common/pom.xml  |  5 ++
 .../org/apache/pulsar/common/api/Commands.java |  6 +--
 .../pulsar/common/util/collections/Pair.java   | 37 --
 19 files changed, 88 insertions(+), 228 deletions(-)

diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCache.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCache.java
index 33a16cb..bb8f2a6 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCache.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCache.java
@@ -21,7 +21,7 @@ package org.apache.bookkeeper.mledger.impl;
 import org.apache.bookkeeper.client.LedgerHandle;
 import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback;
 import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback;
-import org.apache.bookkeeper.mledger.util.Pair;
+import org.apache.commons.lang3.tuple.Pair;
 
 /**
  * Cache of entries used by a single ManagedLedger. An EntryCache is compared 
to other EntryCache instances using their
diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheDefaultEvictionPolicy.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheDefaultEvictionPolicy.java
index fc9e790..682acbb 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheDefaultEvictionPolicy.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheDefaultEvictionPolicy.java
@@ -23,7 +23,7 @@ import static java.util.Collections.reverseOrder;
 
 import com.google.common.collect.Lists;
 import java.util.List;
-import org.apache.bookkeeper.mledger.util.Pair;
+import org.apache.commons.lang3.tuple.Pair;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -85,8 +85,8 @@ public class EntryCacheDefaultEvictionPolicy implements 
EntryCacheEvictionPolicy
 }
 
 Pair evicted = 
entryCache.evictEntries(singleCacheSizeToFree);
-evictedEntries += evicted.first;
-evictedSize += evicted.second;
+evictedEntries += evicted.getLeft();
+evictedSize += evicted.getRight();
 }
 
 log.info("Completed cache eviction. Removed {} entries from {} caches. 
({} Mb)", evictedEntries,
diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
index 60ba634..37ddc54 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
@@ -35,9 +35,9 @@ import org.apache.bookkeeper.client.LedgerHandle;
 import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback;
 import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback;
 import org.apache.bookkeeper.mledger.ManagedLedgerException;
-import org.apache.bookkeeper.mledger.util.Pair;
 import org.apache.bookkeeper.mledger.util.RangeCache;
 import org.apache.bookkeeper.mledger.util.RangeCache.Weighter;
+import org.apache.commons.lang3.tuple.Pair;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -132,8 +132,8 @@ public class EntryCacheImpl implements EntryCache {
 final PositionImpl 

[GitHub] merlimat closed pull request #1515: Consolidate Pair implementations

[GitBox]

merlimat closed pull request #1515: Consolidate Pair implementations
URL: https://github.com/apache/incubator-pulsar/pull/1515
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCache.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCache.java
index 33a16cb839..bb8f2a651f 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCache.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCache.java
@@ -21,7 +21,7 @@
 import org.apache.bookkeeper.client.LedgerHandle;
 import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback;
 import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback;
-import org.apache.bookkeeper.mledger.util.Pair;
+import org.apache.commons.lang3.tuple.Pair;
 
 /**
  * Cache of entries used by a single ManagedLedger. An EntryCache is compared 
to other EntryCache instances using their
diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheDefaultEvictionPolicy.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheDefaultEvictionPolicy.java
index fc9e790656..682acbba2b 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheDefaultEvictionPolicy.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheDefaultEvictionPolicy.java
@@ -23,7 +23,7 @@
 
 import com.google.common.collect.Lists;
 import java.util.List;
-import org.apache.bookkeeper.mledger.util.Pair;
+import org.apache.commons.lang3.tuple.Pair;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -85,8 +85,8 @@ public void doEviction(List caches, long 
sizeToFree) {
 }
 
 Pair evicted = 
entryCache.evictEntries(singleCacheSizeToFree);
-evictedEntries += evicted.first;
-evictedSize += evicted.second;
+evictedEntries += evicted.getLeft();
+evictedSize += evicted.getRight();
 }
 
 log.info("Completed cache eviction. Removed {} entries from {} caches. 
({} Mb)", evictedEntries,
diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
index 60ba634e14..37ddc54629 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
@@ -35,9 +35,9 @@
 import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback;
 import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback;
 import org.apache.bookkeeper.mledger.ManagedLedgerException;
-import org.apache.bookkeeper.mledger.util.Pair;
 import org.apache.bookkeeper.mledger.util.RangeCache;
 import org.apache.bookkeeper.mledger.util.RangeCache.Weighter;
+import org.apache.commons.lang3.tuple.Pair;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -132,8 +132,8 @@ public void invalidateEntries(final PositionImpl 
lastPosition) {
 final PositionImpl firstPosition = PositionImpl.get(-1, 0);
 
 Pair removed = entries.removeRange(firstPosition, 
lastPosition, true);
-int entriesRemoved = removed.first;
-long sizeRemoved = removed.second;
+int entriesRemoved = removed.getLeft();
+long sizeRemoved = removed.getRight();
 if (log.isDebugEnabled()) {
 log.debug("[{}] Invalidated entries up to {} - Entries removed: {} 
- Size removed: {}", ml.getName(),
 lastPosition, entriesRemoved, sizeRemoved);
@@ -148,8 +148,8 @@ public void invalidateAllEntries(long ledgerId) {
 final PositionImpl lastPosition = PositionImpl.get(ledgerId + 1, 0);
 
 Pair removed = entries.removeRange(firstPosition, 
lastPosition, false);
-int entriesRemoved = removed.first;
-long sizeRemoved = removed.second;
+int entriesRemoved = removed.getLeft();
+long sizeRemoved = removed.getRight();
 if (log.isDebugEnabled()) {
 log.debug("[{}] Invalidated all entries on ledger {} - Entries 
removed: {} - Size removed: {}",
 ml.getName(), ledgerId, entriesRemoved, sizeRemoved);
@@ -299,8 +299,8 @@ public int compareTo(EntryCache other) {
 public Pair evictEntries(long sizeToFree) {
 checkArgument(sizeToFree > 0);
 Pair evicted = 
entries.evictLeastAccessedEntries(sizeToFree);
-int evictedEntries = evicted.first;
-long evictedSize = evicted.second;
+

[GitHub] merlimat opened a new pull request #1516: Added test to ensure property name can contain '_' characters.

[GitBox]

merlimat opened a new pull request #1516: Added test to ensure property name 
can contain '_' characters.
URL: https://github.com/apache/incubator-pulsar/pull/1516
 
 
   ### Motivation
   
   As reported in #1506, in 1.22 we still have inconsistencies between what's 
allowed as a valid property name and what is allowed when creating the 
namespace. 
   
   This issue was already fixed in master. `_` are allowed in property name and 
invalid characters are correctly rejected when creating the property.
   
   Adding here tests to ensure the behavior doesn't change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mgodave opened a new pull request #1515: Consolidate Pair implementations

[GitBox]

mgodave opened a new pull request #1515: Consolidate Pair implementations
URL: https://github.com/apache/incubator-pulsar/pull/1515
 
 
   ### Motivation
   
   While working in the codebase I noticed that there were at least three 
different implementations of a "Pair" type being used. I decided to 
consolidate them into one implementation from commons-lang3. Please advise if 
it is somehow desirable to keep one of both of the deleted implementations.
   
   ### Modifications
   
   Replace:
   org.apache.bookkeeper.mledger.util.Pair
   org.apache.pulsar.common.util.collections.Pair
   
   With:
   org.apache.commons.lang3.tuple.Pair
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1513: Managed ledger uses ReadHandle in read path

[GitBox]

merlimat commented on a change in pull request #1513: Managed ledger uses 
ReadHandle in read path
URL: https://github.com/apache/incubator-pulsar/pull/1513#discussion_r179807644
 
 

 ##
 File path: 
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
 ##
 @@ -239,43 +243,44 @@ public void asyncReadEntry(LedgerHandle lh, long 
firstEntry, long lastEntry, boo
 }
 
 // Read all the entries from bookkeeper
-lh.asyncReadEntries(firstEntry, lastEntry, (rc, lh1, sequence, cb) 
-> {
-
-if (rc != BKException.Code.OK) {
-if (rc == BKException.Code.TooManyRequestsException) {
-
callback.readEntriesFailed(createManagedLedgerException(rc), ctx);
-} else {
-ml.invalidateLedgerHandle(lh1, rc);
-ManagedLedgerException mlException = 
createManagedLedgerException(rc);
-callback.readEntriesFailed(mlException, ctx);
-}
-return;
-}
-
-checkNotNull(ml.getName());
-checkNotNull(ml.getExecutor());
-ml.getExecutor().executeOrdered(ml.getName(), safeRun(() -> {
-// We got the entries, we need to transform them to a 
List<> type
-long totalSize = 0;
-final List entriesToReturn = 
Lists.newArrayListWithExpectedSize(entriesToRead);
-while (sequence.hasMoreElements()) {
-// Insert the entries at the end of the list (they 
will be unsorted for now)
-LedgerEntry ledgerEntry = sequence.nextElement();
-EntryImpl entry = EntryImpl.create(ledgerEntry);
-ledgerEntry.getEntryBuffer().release();
-
-entriesToReturn.add(entry);
-
-totalSize += entry.getLength();
-
-}
-
-
manager.mlFactoryMBean.recordCacheMiss(entriesToReturn.size(), totalSize);
-ml.getMBean().addReadEntriesSample(entriesToReturn.size(), 
totalSize);
-
-callback.readEntriesComplete((List) entriesToReturn, ctx);
-}));
-}, callback);
+lh.readAsync(firstEntry, lastEntry).whenComplete(
+(ledgerEntries, exception) -> {
+if (exception != null) {
+if (exception instanceof BKException
+&& ((BKException)exception).getCode() == 
BKException.Code.TooManyRequestsException) {
+
callback.readEntriesFailed(createManagedLedgerException(exception), ctx);
+} else {
+ml.invalidateLedgerHandle(lh, exception);
+ManagedLedgerException mlException = 
createManagedLedgerException(exception);
+callback.readEntriesFailed(mlException, ctx);
+}
+return;
+}
+
+checkNotNull(ml.getName());
+checkNotNull(ml.getExecutor());
+ml.getExecutor().executeOrdered(ml.getName(), 
safeRun(() -> {
 
 Review comment:
   As you did in some other part of code, even here we could use 
`whenCompleteAsync()` with the `ml.executor().chooseThread(ml.getName()`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mgodave commented on a change in pull request #1505: Proxy admin web requests

[GitBox]

mgodave commented on a change in pull request #1505: Proxy admin web requests
URL: https://github.com/apache/incubator-pulsar/pull/1505#discussion_r179798442
 
 

 ##
 File path: pulsar-proxy/pom.xml
 ##
 @@ -70,6 +70,11 @@
   jetty-servlets
 
 
+
+  org.eclipse.jetty
+  jetty-proxy
 
 Review comment:
   I spent a day or two following another approach until I had enough and went 
searching for a better answer, this seemed to fit the bill perfectly. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mgodave commented on a change in pull request #1505: Proxy admin web requests

[GitBox]

mgodave commented on a change in pull request #1505: Proxy admin web requests
URL: https://github.com/apache/incubator-pulsar/pull/1505#discussion_r179798302
 
 

 ##
 File path: 
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
 ##
 @@ -0,0 +1,83 @@
+package org.apache.pulsar.proxy.server;
+
+import java.io.IOException;
+import java.security.cert.X509Certificate;
+import javax.net.ssl.SSLContext;
+import org.apache.pulsar.client.api.Authentication;
+import org.apache.pulsar.client.api.AuthenticationDataProvider;
+import org.apache.pulsar.client.api.AuthenticationFactory;
+import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.common.util.SecurityUtility;
+import org.eclipse.jetty.client.HttpClient;
+import org.eclipse.jetty.proxy.ProxyServlet;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class AdminProxyHandler extends ProxyServlet.Transparent {
+private static final Logger LOG = 
LoggerFactory.getLogger(AdminProxyHandler.class);
+
+private final ProxyConfiguration config;
+
+public AdminProxyHandler(ProxyConfiguration config) {
+this.config = config;
+}
+
+@Override
+protected HttpClient newHttpClient() {
+try {
+Authentication auth = AuthenticationFactory.create(
+config.getBrokerClientAuthenticationPlugin(),
+config.getBrokerClientAuthenticationParameters()
+);
+
+if (auth != null) {
+auth.start();
+}
+
+boolean useTls = 
config.getBrokerServiceURL().startsWith("https://;);
+
+if (useTls) {
+try {
+X509Certificate trustCertificates[] = SecurityUtility
+
.loadCertificatesFromPemFile(config.getTlsTrustCertsFilePath());
+
+SSLContext sslCtx;
+AuthenticationDataProvider authData = auth.getAuthData();
+if (authData.hasDataForTls()) {
+sslCtx = SecurityUtility.createSslContext(
+config.isTlsAllowInsecureConnection(),
+trustCertificates,
+authData.getTlsCertificates(),
+authData.getTlsPrivateKey()
+);
+} else {
+sslCtx = SecurityUtility.createSslContext(
+config.isTlsAllowInsecureConnection(),
+trustCertificates
+);
+}
+
+SslContextFactory contextFactory = new SslContextFactory();
+contextFactory.setSslContext(sslCtx);
+return new HttpClient(contextFactory);
+} catch (Exception e) {
+try {
+if (auth != null) {
+auth.close();
+}
+} catch (IOException ioe) {
+LOG.error("Failed to close the authentication 
service", ioe);
+}
+throw new 
PulsarClientException.InvalidConfigurationException(e.getMessage());
+}
+}
+} catch (PulsarClientException e) {
+e.printStackTrace();
 
 Review comment:
   I think I just left this becuase I wasn't sure what to do at the time. It 
certainly shouldn't stay like this, let me take a look.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mgodave commented on a change in pull request #1505: Proxy admin web requests

[GitBox]

mgodave commented on a change in pull request #1505: Proxy admin web requests
URL: https://github.com/apache/incubator-pulsar/pull/1505#discussion_r179798123
 
 

 ##
 File path: pulsar-proxy/pom.xml
 ##
 @@ -70,6 +70,11 @@
   jetty-servlets
 
 
+
+  org.eclipse.jetty
+  jetty-proxy
 
 Review comment:
   It should, I've explicitly added the option but I will test this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mgodave commented on a change in pull request #1505: Proxy admin web requests

[GitBox]

mgodave commented on a change in pull request #1505: Proxy admin web requests
URL: https://github.com/apache/incubator-pulsar/pull/1505#discussion_r179798017
 
 

 ##
 File path: 
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyServiceStarter.java
 ##
 @@ -121,6 +121,7 @@ public void run() {
 server.addServlet("/metrics", new ServletHolder(MetricsServlet.class));
 server.addRestResources("/", VipStatus.class.getPackage().getName(),
 VipStatus.ATTRIBUTE_STATUS_FILE_PATH, 
config.getStatusFilePath());
+server.addServlet("/admin", new ServletHolder(new 
AdminProxyHandler(config)));
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1399: Integration smoke test for pulsar

[GitBox]

merlimat closed pull request #1399: Integration smoke test for pulsar
URL: https://github.com/apache/incubator-pulsar/pull/1399
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/tests/integration/pom.xml b/tests/integration/pom.xml
new file mode 100644
index 00..a2e36d4556
--- /dev/null
+++ b/tests/integration/pom.xml
@@ -0,0 +1,38 @@
+
+
+http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd; 
xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+  pom
+  4.0.0
+  
+org.apache.pulsar.tests
+tests-parent
+2.0.0-incubating-SNAPSHOT
+  
+
+  org.apache.pulsar.tests
+  integration
+  Apache Pulsar :: Tests :: Integration
+  
+smoke
+  
+
diff --git a/tests/integration/smoke/pom.xml b/tests/integration/smoke/pom.xml
new file mode 100644
index 00..eb3d49911d
--- /dev/null
+++ b/tests/integration/smoke/pom.xml
@@ -0,0 +1,47 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="
+  http://maven.apache.org/POM/4.0.0
+  http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  4.0.0
+  
+org.apache.pulsar.tests
+integration-tests-base
+2.0.0-incubating-SNAPSHOT
+../../integration-tests-base
+  
+
+  org.apache.pulsar.tests.integration
+  smoke
+  jar
+  Apache Pulsar :: Tests :: Integration Tests :: Smoke test
+
+  
+
+  org.apache.pulsar
+  pulsar-client
+  ${project.version}
+
+  
+
diff --git 
a/tests/integration/smoke/src/test/java/org/apache/pulsar/tests/integration/TestSmoke.java
 
b/tests/integration/smoke/src/test/java/org/apache/pulsar/tests/integration/TestSmoke.java
new file mode 100644
index 00..6ecd0536ad
--- /dev/null
+++ 
b/tests/integration/smoke/src/test/java/org/apache/pulsar/tests/integration/TestSmoke.java
@@ -0,0 +1,98 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements.  See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership.  The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.pulsar.tests.integration;
+
+import com.github.dockerjava.api.DockerClient;
+
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.tests.DockerUtils;
+import org.apache.pulsar.tests.PulsarClusterUtils;
+
+import org.jboss.arquillian.testng.Arquillian;
+import org.jboss.arquillian.test.api.ArquillianResource;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestSmoke extends Arquillian {
+private static final Logger LOG = LoggerFactory.getLogger(TestSmoke.class);
+private static byte[] PASSWD = "foobar".getBytes();
+private static String clusterName = "test";
+
+@ArquillianResource
+DockerClient docker;
+
+@Test
+public void testPublishAndConsume() throws Exception {
+Assert.assertTrue(PulsarClusterUtils.startAllBrokers(docker, 
clusterName));
+Assert.assertTrue(PulsarClusterUtils.startAllProxies(docker, 
clusterName));
+
+// create property and namespace
+

[incubator-pulsar] branch master updated: Integration smoke test for pulsar (#1399)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 013ea05  Integration smoke test for pulsar (#1399)
013ea05 is described below

commit 013ea058a74675764d477d4918bce5bd0a433fb9
Author: Ivan Kelly 
AuthorDate: Fri Apr 6 16:21:02 2018 +0200

Integration smoke test for pulsar (#1399)

* Integration smoke test for pulsar

Boots a cluster, sends and receives some messages, shuts down the
cluster.

If anything fails, then something is very broken.

* license fix

* Disable AnnotationListener for integration tests

The annotation listener adds timeouts to all tests, which changes the
threading model, which breaks arquillian.

I've created an issue[1] on arquillian for this, but until that is fixed,
timeouts cannot be used with arquillian.

[1] https://github.com/arquillian/arquillian-core/issues/168
---
 tests/{ => integration}/pom.xml| 25 ++
 tests/{ => integration/smoke}/pom.xml  | 46 +-
 .../apache/pulsar/tests/integration/TestSmoke.java | 98 ++
 .../smoke/src/test/resources/arquillian.xml| 32 +++
 tests/pom.xml  |  1 +
 5 files changed, 158 insertions(+), 44 deletions(-)

diff --git a/tests/pom.xml b/tests/integration/pom.xml
similarity index 68%
copy from tests/pom.xml
copy to tests/integration/pom.xml
index d4d3137..a2e36d4 100644
--- a/tests/pom.xml
+++ b/tests/integration/pom.xml
@@ -24,28 +24,15 @@
   pom
   4.0.0
   
-org.apache.pulsar
-pulsar
+org.apache.pulsar.tests
+tests-parent
 2.0.0-incubating-SNAPSHOT
   
+
   org.apache.pulsar.tests
-  tests-parent
-  Apache Pulsar :: Tests
+  integration
+  Apache Pulsar :: Tests :: Integration
   
-docker-images
-integration-tests-utils
-integration-tests-topologies
-integration-tests-base
+smoke
   
-  
-
-  
-org.apache.maven.plugins
-maven-deploy-plugin
-
-  true
-
-  
-
-  
 
diff --git a/tests/pom.xml b/tests/integration/smoke/pom.xml
similarity index 52%
copy from tests/pom.xml
copy to tests/integration/smoke/pom.xml
index d4d3137..eb3d499 100644
--- a/tests/pom.xml
+++ b/tests/integration/smoke/pom.xml
@@ -19,33 +19,29 @@
 under the License.
 
 -->
-http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd; 
xmlns="http://maven.apache.org/POM/4.0.0;
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
-  pom
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="
+  http://maven.apache.org/POM/4.0.0
+  http://maven.apache.org/xsd/maven-4.0.0.xsd;>
   4.0.0
   
-org.apache.pulsar
-pulsar
+org.apache.pulsar.tests
+integration-tests-base
 2.0.0-incubating-SNAPSHOT
+../../integration-tests-base
   
-  org.apache.pulsar.tests
-  tests-parent
-  Apache Pulsar :: Tests
-  
-docker-images
-integration-tests-utils
-integration-tests-topologies
-integration-tests-base
-  
-  
-
-  
-org.apache.maven.plugins
-maven-deploy-plugin
-
-  true
-
-  
-
-  
+
+  org.apache.pulsar.tests.integration
+  smoke
+  jar
+  Apache Pulsar :: Tests :: Integration Tests :: Smoke test
+
+  
+
+  org.apache.pulsar
+  pulsar-client
+  ${project.version}
+
+  
 
diff --git 
a/tests/integration/smoke/src/test/java/org/apache/pulsar/tests/integration/TestSmoke.java
 
b/tests/integration/smoke/src/test/java/org/apache/pulsar/tests/integration/TestSmoke.java
new file mode 100644
index 000..6ecd053
--- /dev/null
+++ 
b/tests/integration/smoke/src/test/java/org/apache/pulsar/tests/integration/TestSmoke.java
@@ -0,0 +1,98 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements.  See the 

[GitHub] ivankelly commented on issue #1512: Also log bookkeeper (and managed ledger) in tests

[GitBox]

ivankelly commented on issue #1512: Also log bookkeeper (and managed ledger) in 
tests
URL: https://github.com/apache/incubator-pulsar/pull/1512#issuecomment-379259622
 
 
   retest this please // 
BrokerClientIntegrationTest.testUnsupportedBatchMessageConsumer


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on issue #1513: Managed ledger uses ReadHandle in read path

[GitBox]

ivankelly commented on issue #1513: Managed ledger uses ReadHandle in read path
URL: https://github.com/apache/incubator-pulsar/pull/1513#issuecomment-379259720
 
 
   retest this please // BrokerBkEnsemblesTests.testSkipCorruptDataLedger


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #1514: Compaction allows keyless messages to pass through

[GitBox]

ivankelly opened a new pull request #1514: Compaction allows keyless messages 
to pass through
URL: https://github.com/apache/incubator-pulsar/pull/1514
 
 
   If a message has no key it's impossible to know if a later message
   supercedes it, so in this case, it should be simply passed through
   compaction (i.e. it should appear when reading from a compacted
   topic).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on issue #1512: Also log bookkeeper (and managed ledger) in tests

[GitBox]

ivankelly commented on issue #1512: Also log bookkeeper (and managed ledger) in 
tests
URL: https://github.com/apache/incubator-pulsar/pull/1512#issuecomment-379243136
 
 
   retest this please // ReplicatorTest.testCloseReplicatorStartProducer


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on issue #1501: Rest endpoint to query compaction status

[GitBox]

ivankelly commented on issue #1501: Rest endpoint to query compaction status
URL: https://github.com/apache/incubator-pulsar/pull/1501#issuecomment-379243015
 
 
   retest this please // 
PersistentQueueE2ETest.testCancelReadRequestOnLastDisconnect


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #1513: Managed ledger uses ReadHandle in read path

[GitBox]

ivankelly opened a new pull request #1513: Managed ledger uses ReadHandle in 
read path
URL: https://github.com/apache/incubator-pulsar/pull/1513
 
 
   BookKeeper 4.6 introduced a new API for reading, called
   ReadHandle. This API is an small interface, unlike LedgerHandle, so it
   is possible to provide implementations of ReadHandle that read from
   sources other than bookkeeper, while presenting the same interface as
   when reading from bookkeeper.
   
   This patch changes the managed ledger entry read path to use
   ReadHandle. This is rquired to implement tiered storage (PIP-17).
   
   Master issue: #1511


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zubchenok commented on issue #1506: Underscore `_` should be allowed as property name

[GitBox]

zubchenok commented on issue #1506: Underscore `_` should be allowed as 
property name
URL: 
https://github.com/apache/incubator-pulsar/issues/1506#issuecomment-379233042
 
 
   Before closing the issue, run the unit test against 1.22 release to double 
check that the unit test reproduces the issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zubchenok commented on issue #1506: Underscore `_` should be allowed as property name

[GitBox]

zubchenok commented on issue #1506: Underscore `_` should be allowed as 
property name
URL: 
https://github.com/apache/incubator-pulsar/issues/1506#issuecomment-379232768
 
 
   I use 1.22 release. So seems it is already fixed in the current source.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #1512: Also log bookkeeper (and managed ledger) in tests

[GitBox]

ivankelly opened a new pull request #1512: Also log bookkeeper (and managed 
ledger) in tests
URL: https://github.com/apache/incubator-pulsar/pull/1512
 
 
   Root level is warn, so we need to explicitly set org.apache.bookkeeper
   to info to see the logs.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new issue #1511: PIP-17 : Tiered storage (Master Issue)

[GitBox]

ivankelly opened a new issue #1511: PIP-17 : Tiered storage (Master Issue)
URL: https://github.com/apache/incubator-pulsar/issues/1511
 
 
   Storing backlogs on bookies for a long time can get expensive. If
   there are other cheaper forms of storage available (S3/HDFS), capex
   can be reduced by moving older data to this storage.
   
   The proposed design proposes an interface to do this, with an initial
   implementation targeted at S3. However the interface is general enough
   that HDFS could also be used.
   
   Design: 
https://github.com/apache/incubator-pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics
   Mailing List discussion: 
https://lists.apache.org/thread.html/3f1c98658395c92e3c858ff966aa829dba0e165a28ef350e0ec3a33f@%3Cdev.pulsar.apache.org%3E
   Prototype: 
https://github.com/ivankelly/incubator-pulsar/tree/s3-offload-proto 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on issue #1399: Integration smoke test for pulsar

[GitBox]

ivankelly commented on issue #1399: Integration smoke test for pulsar
URL: https://github.com/apache/incubator-pulsar/pull/1399#issuecomment-379208812
 
 
   finally


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on issue #1501: Rest endpoint to query compaction status

[GitBox]

ivankelly commented on issue #1501: Rest endpoint to query compaction status
URL: https://github.com/apache/incubator-pulsar/pull/1501#issuecomment-379204955
 
 
   retest this please // flake in c++ AuthPluginTest.testAthenz


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on issue #1399: Integration smoke test for pulsar

[GitBox]

ivankelly commented on issue #1399: Integration smoke test for pulsar
URL: https://github.com/apache/incubator-pulsar/pull/1399#issuecomment-379194411
 
 
   retest this please 
   
   // flake on 
PersistentFailoverE2ETest.testSimpleConsumerEventsWithoutPartition (java8)
   // flake on ConsumerStatsTest.testCachingMechanism (c++/python)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie opened a new issue #1510: java.lang.IllegalArgumentException on refreshing stats

[GitBox]

sijie opened a new issue #1510: java.lang.IllegalArgumentException on 
refreshing stats
URL: https://github.com/apache/incubator-pulsar/issues/1510
 
 
   Found following stacktrace when running `AdminApiTest#partitionedTopics`
   
   ```
   00:55:33.359 
[bookkeeper-ml-scheduler-OrderedScheduler-12-0:org.apache.bookkeeper.common.util.SafeRunnable@38]
 ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable 
caught
   java.lang.IllegalArgumentException: null
   at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:108) 
~[guava-20.0.jar:?]
   at 
org.apache.bookkeeper.mledger.util.Rate.calculateRate(Rate.java:60) 
~[managed-ledger-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]
   at 
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryMBeanImpl.refreshStats(ManagedLedgerFactoryMBeanImpl.java:41)
 ~[managed-ledger-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]
   at 
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.refreshStats(ManagedLedgerFactoryImpl.java:147)
 ~[managed-ledger-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]
   at 
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$new$2(ManagedLedgerFactoryImpl.java:140)
 ~[managed-ledger-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]
   at 
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) 
[bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
   at 
org.apache.bookkeeper.shaded.com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.run(MoreExecutors.java:587)
 [bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
   at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_144]
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[?:1.8.0_144]
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [?:1.8.0_144]
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [?:1.8.0_144]
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_144]
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_144]
   at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 [netty-all-4.1.21.Final.jar:4.1.21.Final]
   at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services