[GitHub] sijie closed pull request #2065: GCS offload support(2): replace `s3client` api with `jclouds` related api

[GitBox]

sijie closed pull request #2065: GCS offload support(2): replace `s3client` api 
with `jclouds` related api 
URL: https://github.com/apache/incubator-pulsar/pull/2065
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/conf/broker.conf b/conf/broker.conf
index 2e40d82a87..96433aa517 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -486,7 +486,7 @@ 
schemaRegistryStorageClassName=org.apache.pulsar.broker.service.schema.Bookkeepe
 
 ### --- Ledger Offloading --- ###
 
-# Driver to use to offload old data to long term storage (Possible values: S3)
+# Driver to use to offload old data to long term storage (Possible values: S3, 
aws-s3, google-cloud-storage)
 managedLedgerOffloadDriver=
 
 # Maximum number of thread pool threads for ledger offloading
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt 
b/distribution/server/src/assemble/LICENSE.bin.txt
index 33b97481da..85a97e5441 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -334,10 +334,10 @@ The Apache Software License, Version 2.0
 - io.swagger-swagger-annotations-1.5.3.jar
 - io.swagger-swagger-core-1.5.3.jar
 - io.swagger-swagger-models-1.5.3.jar
- * DataSketches 
+ * DataSketches
 - com.yahoo.datasketches-memory-0.8.3.jar
 - com.yahoo.datasketches-sketches-core-0.8.3.jar
- * Apache Commons 
+ * Apache Commons
 - commons-beanutils-commons-beanutils-1.7.0.jar
 - commons-beanutils-commons-beanutils-core-1.8.0.jar
 - commons-cli-commons-cli-1.2.jar
@@ -461,6 +461,12 @@ The Apache Software License, Version 2.0
 - org.xerial.snappy-snappy-java-1.1.1.3.jar
   * Flatbuffers Java
 - com.google.flatbuffers-flatbuffers-java-1.9.0.jar
+  * Apache Jclouds
+- org.apache.jclouds-allblobstore-2.2.0-SNAPSHOT.jar
+  * Google Guice Core Library
+- com.google.inject.guice-3.0.jar
+- com.google.inject.extensions:guice-multibindings-3.0.jar
+- com.google.inject.extensions:guice-assistedinject-3.0.jar
 
 
 BSD 3-clause "New" or "Revised" License
diff --git a/jclouds-shaded/pom.xml b/jclouds-shaded/pom.xml
new file mode 100644
index 00..8aa1786c90
--- /dev/null
+++ b/jclouds-shaded/pom.xml
@@ -0,0 +1,105 @@
+
+
+http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+  xmlns="http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+  4.0.0
+  
+org.apache.pulsar
+pulsar
+2.2.0-incubating-SNAPSHOT
+..
+  
+
+  jclouds-shaded
+  Apache Pulsar :: Jclouds shaded
+
+  
+
+  com.google.code.gson
+  gson
+  2.5
+
+
+  org.apache.jclouds
+  jclouds-allblobstore
+  2.2.0-SNAPSHOT
+
+  
+
+  
+
+  jclouds-snapshots
+  https://repository.apache.org/content/repositories/snapshots
+  
+true
+  
+
+  
+
+  
+
+  
+org.apache.maven.plugins
+maven-shade-plugin
+
+  
+package
+
+  shade
+
+
+  true
+  
true
+  false
+
+  
+
+  com.google.code.gson:gson
+  com.google.guava:guava
+  org.apache.jclouds:*
+  org.apache.jclouds.api:*
+  org.apache.jclouds.common:*
+  org.apache.jclouds.provider:*
+  
com.google.inject.extensions:guice-assistedinject
+  com.google.inject:guice
+  
com.google.inject.extensions:guice-multibindings
+
+  
+
+  
+
+  com.google
+  
org.apache.pulsar.shaded.com.google
+
+  
+  
+
+
+  
+
+  
+
+  
+
+  
+
diff --git a/pom.xml b/pom.xml
index a6d645379b..88c2591ad9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -100,6 +100,8 @@ flexible messaging model and an intuitive client 
API.
 pulsar-zookeeper
 pulsar-log4j2-appender
 protobuf-shaded
+
+jclouds-shaded
 
 
 pulsar-functions
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index d505c7a881..ac264bdf5b 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -127,7 +127,7 @@
 
 
   com.amazonaws
-  aws-java-sdk-s3
+  aws-java-sdk-core
 
 
 
@@ -273,6 +273,12 @@
   ${project.version}
   test
 
+
+
+  org.apache.pulsar
+  jclouds-shaded
+  ${project.version}
+
   
 
   
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java 

[incubator-pulsar] branch master updated: GCS offload support(2): replace `s3client` api with `jclouds` related api (#2065)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 3c8d13c  GCS offload support(2): replace `s3client` api with `jclouds` 
related api  (#2065)
3c8d13c is described below

commit 3c8d13c0546b7fed0100d499bd1841f7ce2a127e
Author: Jia Zhai 
AuthorDate: Fri Jul 20 13:27:05 2018 +0800

GCS offload support(2): replace `s3client` api with `jclouds` related api  
(#2065)

This is the second part to support `Google Cloud Storage` offload.
It aims to replace "s3 client" api with "jclouds" api, and make sure unit 
test and integration test passed.
There will be a following change to add `Google Cloud Storage` support and 
related test.

change:
replace `s3client` api with `jclouds` related api in 
`S3ManagedLedgerOffloader`

Master Issue: #2067
---
 conf/broker.conf   |   2 +-
 distribution/server/src/assemble/LICENSE.bin.txt   |  10 +-
 jclouds-shaded/pom.xml | 105 +++
 pom.xml|   2 +
 pulsar-broker/pom.xml  |   8 +-
 .../org/apache/pulsar/broker/PulsarService.java|   6 +-
 ...pl.java => BlobStoreBackedInputStreamImpl.java} |  63 ++--
 ...mpl.java => BlobStoreBackedReadHandleImpl.java} |  49 ++-
 ...r.java => BlobStoreManagedLedgerOffloader.java} | 220 +-
 ...st.java => BlobStoreBackedInputStreamTest.java} | 114 ---
 .../{S3TestBase.java => BlobStoreTestBase.java}|  36 ++-
 .../org/apache/pulsar/broker/offload/S3Mock.java   | 334 -
 ...va => BlobStoreManagedLedgerOffloaderTest.java} | 190 ++--
 ...luster-2-bookie-1-broker-unstarted-with-s3.yaml |   4 +-
 14 files changed, 518 insertions(+), 625 deletions(-)

diff --git a/conf/broker.conf b/conf/broker.conf
index 2e40d82..96433aa 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -486,7 +486,7 @@ 
schemaRegistryStorageClassName=org.apache.pulsar.broker.service.schema.Bookkeepe
 
 ### --- Ledger Offloading --- ###
 
-# Driver to use to offload old data to long term storage (Possible values: S3)
+# Driver to use to offload old data to long term storage (Possible values: S3, 
aws-s3, google-cloud-storage)
 managedLedgerOffloadDriver=
 
 # Maximum number of thread pool threads for ledger offloading
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt 
b/distribution/server/src/assemble/LICENSE.bin.txt
index 33b9748..85a97e5 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -334,10 +334,10 @@ The Apache Software License, Version 2.0
 - io.swagger-swagger-annotations-1.5.3.jar
 - io.swagger-swagger-core-1.5.3.jar
 - io.swagger-swagger-models-1.5.3.jar
- * DataSketches 
+ * DataSketches
 - com.yahoo.datasketches-memory-0.8.3.jar
 - com.yahoo.datasketches-sketches-core-0.8.3.jar
- * Apache Commons 
+ * Apache Commons
 - commons-beanutils-commons-beanutils-1.7.0.jar
 - commons-beanutils-commons-beanutils-core-1.8.0.jar
 - commons-cli-commons-cli-1.2.jar
@@ -461,6 +461,12 @@ The Apache Software License, Version 2.0
 - org.xerial.snappy-snappy-java-1.1.1.3.jar
   * Flatbuffers Java
 - com.google.flatbuffers-flatbuffers-java-1.9.0.jar
+  * Apache Jclouds
+- org.apache.jclouds-allblobstore-2.2.0-SNAPSHOT.jar
+  * Google Guice Core Library
+- com.google.inject.guice-3.0.jar
+- com.google.inject.extensions:guice-multibindings-3.0.jar
+- com.google.inject.extensions:guice-assistedinject-3.0.jar
 
 
 BSD 3-clause "New" or "Revised" License
diff --git a/jclouds-shaded/pom.xml b/jclouds-shaded/pom.xml
new file mode 100644
index 000..8aa1786
--- /dev/null
+++ b/jclouds-shaded/pom.xml
@@ -0,0 +1,105 @@
+
+
+http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+  xmlns="http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+  4.0.0
+  
+org.apache.pulsar
+pulsar
+2.2.0-incubating-SNAPSHOT
+..
+  
+
+  jclouds-shaded
+  Apache Pulsar :: Jclouds shaded
+
+  
+
+  com.google.code.gson
+  gson
+  2.5
+
+
+  org.apache.jclouds
+  jclouds-allblobstore
+  2.2.0-SNAPSHOT
+
+  
+
+  
+
+  jclouds-snapshots
+  https://repository.apache.org/content/repositories/snapshots
+  
+true
+  
+
+  
+
+  
+
+  
+org.apache.maven.plugins
+maven-shade-plugin
+
+  
+package
+
+  shade
+
+
+  true
+  
true
+  false
+
+  
+
+  com.google.code.gson:gson
+  com.google.guava:guava
+  

[GitHub] sijie commented on issue #2208: Fix License Issue

[GitBox]

sijie commented on issue #2208: Fix License Issue
URL: https://github.com/apache/incubator-pulsar/pull/2208#issuecomment-406490095
 
 
   are you running against latest master? latest master has the problem. my 
change should be fixing that. did you test my branch @jiazhai ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar.wiki] branch master updated: Updated Home (markdown)

[jerrypeng]

This is an automated email from the ASF dual-hosted git repository.

jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 0a2f109  Updated Home (markdown)
0a2f109 is described below

commit 0a2f109845f85451126839904f78003826487dcd
Author: Boyang Jerry Peng 
AuthorDate: Thu Jul 19 22:02:31 2018 -0700

Updated Home (markdown)
---
 Home.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Home.md b/Home.md
index 0c15b9e..a2216a5 100644
--- a/Home.md
+++ b/Home.md
@@ -5,7 +5,7 @@
 
 We encourage to document any big change or feature or any addition to public 
user APIs through a design document to be discussed with the community.
 
-* [PIP 18: Pulsar 
SQL](https://github.com/apache/incubator-pulsar/wiki/PIP-18:-Pulsar-SQL)
+* [PIP 19: Pulsar 
SQL](https://github.com/apache/incubator-pulsar/wiki/PIP-19:-Pulsar-SQL)
 * [PIP 17: Tiered storage for Pulsar 
topics](https://github.com/apache/incubator-pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics)
 * [PIP 16: Pulsar "instance" terminology 
change](https://github.com/apache/incubator-pulsar/wiki/PIP-16:-Pulsar-%22instance%22-terminology-change)
 * [PIP 15: Pulsar 
Functions](https://github.com/apache/incubator-pulsar/wiki/PIP-15:-Pulsar-Functions)

[incubator-pulsar.wiki] branch master updated: Updated PIP 18: Pulsar SQL (markdown)

[jerrypeng]

This is an automated email from the ASF dual-hosted git repository.

jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new e2d2002  Updated PIP 18: Pulsar SQL (markdown)
e2d2002 is described below

commit e2d2002063d5afe3262015fde33ca978cb5ed01b
Author: Boyang Jerry Peng 
AuthorDate: Thu Jul 19 22:02:10 2018 -0700

Updated PIP 18: Pulsar SQL (markdown)
---
 PIP-18:-Pulsar-SQL.md => PIP-19:-Pulsar-SQL.md | 0
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/PIP-18:-Pulsar-SQL.md b/PIP-19:-Pulsar-SQL.md
similarity index 100%
rename from PIP-18:-Pulsar-SQL.md
rename to PIP-19:-Pulsar-SQL.md

[GitHub] jiazhai commented on issue #2208: Fix License Issue

[GitBox]

jiazhai commented on issue #2208: Fix License Issue
URL: https://github.com/apache/incubator-pulsar/pull/2208#issuecomment-406462205
 
 
   @sijie Here is the running and output.
   run the command, under pulsar root dir, 
   ```
   sh src/check-binary-license 
./distribution/server/target/apache-pulsar-2.2.0-incubating-SNAPSHOT-bin.tar.gz
   
   
   org.apache.commons-commons-compress-1.15.jar unaccounted for in LICENSE
   org.objenesis-objenesis-2.1.jar unaccounted for in LICENSE
   org.apache.commons-commons-compress-1.8.1.jar mentioned in LICENSE, but not 
bundled
   org.apache.avro-avro-protobuf-1.8.2.jar mentioned in LICENSE, but not bundled
   com.google.flatbuffers-flatbuffers-java-1.9.0.jar mentioned in LICENSE, but 
not bundled
   
   It looks like there are issues with the LICENSE/NOTICE.
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhaijack commented on issue #2205: Issue #2204: support v2 namespace in cpp client

[GitBox]

zhaijack commented on issue #2205: Issue #2204: support v2 namespace in cpp 
client
URL: https://github.com/apache/incubator-pulsar/pull/2205#issuecomment-406457942
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhaijack removed a comment on issue #2205: Issue #2204: support v2 namespace in cpp client

[GitBox]

zhaijack removed a comment on issue #2205: Issue #2204: support v2 namespace in 
cpp client
URL: https://github.com/apache/incubator-pulsar/pull/2205#issuecomment-406457813
 
 
   retest this please 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhaijack commented on issue #2205: Issue #2204: support v2 namespace in cpp client

[GitBox]

zhaijack commented on issue #2205: Issue #2204: support v2 namespace in cpp 
client
URL: https://github.com/apache/incubator-pulsar/pull/2205#issuecomment-406457813
 
 
   retest this please 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2208: Fix License Issue

[GitBox]

sijie commented on issue #2208: Fix License Issue
URL: https://github.com/apache/incubator-pulsar/pull/2208#issuecomment-406457217
 
 
   this tracks the server dist one. where do you run this? and what is your 
output?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jiazhai commented on issue #2208: Fix License Issue

[GitBox]

jiazhai commented on issue #2208: Fix License Issue
URL: https://github.com/apache/incubator-pulsar/pull/2208#issuecomment-406457017
 
 
   @sijie Seems `src/check-binary-license` reported more not bundled libs. This 
is only track the server dist ones?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie opened a new pull request #2209: Exclude helm files from Apache Rat Check.

[GitBox]

sijie opened a new pull request #2209: Exclude helm files from Apache Rat Check.
URL: https://github.com/apache/incubator-pulsar/pull/2209
 
 
   *Motivation*
   
   Some of the helm files (e.g. helm ignore file and template file) doesn't 
have ASF license header.
   It is tricky to add license headers for them. so exclude them from rat check.
   
   *Changes*
   
   Exclude two files from Apache Rat Check
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie opened a new pull request #2208: Fix License Issue

[GitBox]

sijie opened a new pull request #2208: Fix License Issue
URL: https://github.com/apache/incubator-pulsar/pull/2208
 
 
### Motivation
   
   Some libaries are not bundled with server distribution anymore.
   
### Changes
   
   Update the LICENSE.bin.txt to reflect the libraries we have in server 
distribution.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar.wiki] branch master updated: Updated Release Candidate Validation (markdown)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 10c5a16  Updated Release Candidate Validation (markdown)
10c5a16 is described below

commit 10c5a1647a3c0295b9408fc3b3fd032d478b4600
Author: Sijie Guo 
AuthorDate: Thu Jul 19 16:41:41 2018 -0700

Updated Release Candidate Validation (markdown)
---
 Release-Candidate-Validation.md | 5 -
 1 file changed, 5 deletions(-)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index 54111cc..81fd9e2 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -3,7 +3,6 @@
   * [Validate Pub/Sub and Java Functions](#validate-pubsub-and-java-functions)
   * [Validate Connectors](#validate-connectors)
   * [Validate Stateful Functions](#validate-stateful-functions)
-  * [Validate Python Functions](#validate-python-functions)
 - [Validate RPM and DEB packages](#validate-rpm-and-deb-packages)
   * [Prepare](#prepare)
   * [Validate RPM](#validate-rpm)
@@ -524,10 +523,6 @@ value = 10
 value = 20
 ```
 
- Validate Python Functions
-
-
-
 ### Validate RPM and DEB packages
 
  Prepare

[incubator-pulsar.wiki] branch master updated: Updated Release Candidate Validation (markdown)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new a6b5ff6  Updated Release Candidate Validation (markdown)
a6b5ff6 is described below

commit a6b5ff670d227c4a1d3ed28d1771e9fd9e4b7dc4
Author: Sijie Guo 
AuthorDate: Thu Jul 19 16:28:41 2018 -0700

Updated Release Candidate Validation (markdown)
---
 Release-Candidate-Validation.md | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index 033e0cd..54111cc 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -3,12 +3,14 @@
   * [Validate Pub/Sub and Java Functions](#validate-pubsub-and-java-functions)
   * [Validate Connectors](#validate-connectors)
   * [Validate Stateful Functions](#validate-stateful-functions)
+  * [Validate Python Functions](#validate-python-functions)
 - [Validate RPM and DEB packages](#validate-rpm-and-deb-packages)
   * [Prepare](#prepare)
   * [Validate RPM](#validate-rpm)
   * [Validate DEB](#validate-deb)
 
-Here are some instructions for reviewing and validating a release candidate.
+Here are some manual instructions for reviewing and validating a release 
candidate.
+These steps can be automated. Contributions are welcome!
 
 ### Validate the binary distribution
 
@@ -522,6 +524,10 @@ value = 10
 value = 20
 ```
 
+ Validate Python Functions
+
+
+
 ### Validate RPM and DEB packages
 
  Prepare

[incubator-pulsar.wiki] branch master updated: Updated Home (markdown)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 51cae59  Updated Home (markdown)
51cae59 is described below

commit 51cae59b50be8ea2288ff056c60ed2170cd2a5ac
Author: Sijie Guo 
AuthorDate: Thu Jul 19 16:25:20 2018 -0700

Updated Home (markdown)
---
 Home.md | 4 
 1 file changed, 4 insertions(+)

diff --git a/Home.md b/Home.md
index d0d8ebf..0c15b9e 100644
--- a/Home.md
+++ b/Home.md
@@ -24,6 +24,10 @@ We encourage to document any big change or feature or any 
addition to public use
 * [PIP 2: Non Persistent 
topic](https://github.com/apache/incubator-pulsar/wiki/PIP-2:-Non-Persistent-topic)
 * [PIP 1: Pulsar 
Proxy](https://github.com/apache/incubator-pulsar/wiki/PIP-1:-Pulsar-Proxy)
 
+## Contributor documentation
+
+ * [Release Candidate 
Validation](https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation)
+
 ## Committers documentation
 
  * [Create GPG keys to sign release 
artifacts](https://github.com/apache/incubator-pulsar/wiki/Create-GPG-keys-to-sign-release-artifacts)

[GitHub] rdhabalia commented on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

rdhabalia commented on issue #2200: support subscription-type to be passed in 
sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406442736
 
 
   > otherwise we keep finding issues or improvements to do and we never 
actually cut the release :).
   
   alright.
   
   > In general, I would rather have a parameter called "retain ordering " in 
the function configuration,

   changed it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asfgit removed a comment on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

asfgit removed a comment on issue #2200: support subscription-type to be passed 
in sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406437762
 
 
   FAILURE

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Removed the `Context.ack(byte[] messageId)` and expose `Record` instead (#2187)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 6142a4c   Removed the `Context.ack(byte[] messageId)` and expose 
`Record` instead  (#2187)
6142a4c is described below

commit 6142a4c5b03f5a4e26a774fd57ab458844255786
Author: Matteo Merli 
AuthorDate: Thu Jul 19 16:08:16 2018 -0700

 Removed the `Context.ack(byte[] messageId)` and expose `Record` instead  
(#2187)

* Removed the `Context.ack(byte[] messageId)` and expose `Record` instead

* Unified all per-message function context into record

* Fixed compilation

* Fixed compilation

* Fix test

* Fixed mocked object

* Fixed test
---
 .../org/apache/pulsar/functions/api/Context.java   | 20 +
 .../org/apache/pulsar/functions/api/Record.java|  7 ++
 .../pulsar/functions/instance/ContextImpl.java | 89 --
 .../pulsar/functions/instance/JavaInstance.java|  7 +-
 .../functions/instance/JavaInstanceRunnable.java   | 31 
 .../apache/pulsar/functions/sink/PulsarSink.java   |  2 +-
 .../pulsar/functions/source/PulsarRecord.java  |  5 ++
 .../pulsar/functions/source/PulsarSource.java  | 30 ++--
 .../apache/pulsar/functions/windowing/Event.java   | 15 ++--
 .../pulsar/functions/windowing/EventImpl.java  | 12 +--
 .../windowing/WindowFunctionExecutor.java  | 33 
 .../pulsar/functions/windowing/WindowManager.java  | 14 ++--
 .../pulsar/functions/instance/ContextImplTest.java |  7 +-
 .../functions/instance/JavaInstanceTest.java   | 10 +--
 .../windowing/WindowFunctionExecutorTest.java  | 31 ++--
 15 files changed, 164 insertions(+), 149 deletions(-)

diff --git 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
index 2fa513e..9869067 100644
--- 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
+++ 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
@@ -34,17 +34,10 @@ import java.util.concurrent.CompletableFuture;
  */
 public interface Context {
 /**
- * Returns the messageId of the message that we are processing
- * This messageId is a stringified version of the actual MessageId
- * @return the messageId
+ * Access the record associated with the current input value
+ * @return
  */
-byte[] getMessageId();
-
-/**
- * The input topic that the message currently being processed belongs to
- * @return The input topic name
- */
-String getCurrentMessageTopicName();
+Record getCurrentRecord();
 
 /**
  * Get a list of all input topics
@@ -183,11 +176,4 @@ public interface Context {
  */
  CompletableFuture publish(String topicName, O object);
 
-/**
- * By default acknowledgement management is done transparently by Pulsar 
Functions framework.
- * However users can disable that and do ack management by themselves by 
using this API.
- * @param messageId The messageId that needs to be acknowledged
- * @return A future that completes when the framework is done acking the 
message
- */
-CompletableFuture ack(byte[] messageId);
 }
\ No newline at end of file
diff --git 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java
 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java
index 38d6ed3..2704909 100644
--- 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java
+++ 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java
@@ -28,6 +28,13 @@ import java.util.Optional;
 public interface Record {
 
 /**
+ * If the record originated from a topic, report the topic name
+ */
+default Optional getTopicName() {
+return Optional.empty();
+}
+
+/**
  * Return a key if the key has one associated
  */
 Optional getKey();
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
index b41d6f4..8e8f966 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
@@ -18,19 +18,33 @@
  */
 package org.apache.pulsar.functions.instance;
 
+import static com.google.common.base.Preconditions.checkState;
+
 import com.google.gson.Gson;
 import com.google.gson.reflect.TypeToken;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;

[GitHub] merlimat closed pull request #2187: Removed the `Context.ack(byte[] messageId)` and expose `Record` instead

[GitBox]

merlimat closed pull request #2187:  Removed the `Context.ack(byte[] 
messageId)` and expose `Record` instead 
URL: https://github.com/apache/incubator-pulsar/pull/2187
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
index 2fa513e120..98690674b2 100644
--- 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
+++ 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
@@ -34,17 +34,10 @@
  */
 public interface Context {
 /**
- * Returns the messageId of the message that we are processing
- * This messageId is a stringified version of the actual MessageId
- * @return the messageId
+ * Access the record associated with the current input value
+ * @return
  */
-byte[] getMessageId();
-
-/**
- * The input topic that the message currently being processed belongs to
- * @return The input topic name
- */
-String getCurrentMessageTopicName();
+Record getCurrentRecord();
 
 /**
  * Get a list of all input topics
@@ -183,11 +176,4 @@
  */
  CompletableFuture publish(String topicName, O object);
 
-/**
- * By default acknowledgement management is done transparently by Pulsar 
Functions framework.
- * However users can disable that and do ack management by themselves by 
using this API.
- * @param messageId The messageId that needs to be acknowledged
- * @return A future that completes when the framework is done acking the 
message
- */
-CompletableFuture ack(byte[] messageId);
 }
\ No newline at end of file
diff --git 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java
 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java
index 38d6ed36d7..2704909043 100644
--- 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java
+++ 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java
@@ -27,6 +27,13 @@
  */
 public interface Record {
 
+/**
+ * If the record originated from a topic, report the topic name
+ */
+default Optional getTopicName() {
+return Optional.empty();
+}
+
 /**
  * Return a key if the key has one associated
  */
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
index b41d6f48ab..8e8f9665f0 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
@@ -18,19 +18,33 @@
  */
 package org.apache.pulsar.functions.instance;
 
+import static com.google.common.base.Preconditions.checkState;
+
 import com.google.gson.Gson;
 import com.google.gson.reflect.TypeToken;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.TimeUnit;
+
 import lombok.Getter;
 import lombok.Setter;
+
 import org.apache.commons.lang.StringUtils;
-import org.apache.pulsar.client.api.Consumer;
 import org.apache.pulsar.client.api.MessageId;
 import org.apache.pulsar.client.api.Producer;
 import org.apache.pulsar.client.api.ProducerConfiguration;
 import org.apache.pulsar.client.api.PulsarClient;
 import org.apache.pulsar.client.api.PulsarClientException;
-import org.apache.pulsar.client.impl.MultiTopicsConsumerImpl;
 import org.apache.pulsar.functions.api.Context;
+import org.apache.pulsar.functions.api.Record;
 import org.apache.pulsar.functions.api.SerDe;
 import org.apache.pulsar.functions.api.utils.DefaultSerDe;
 import org.apache.pulsar.functions.instance.state.StateContextImpl;
@@ -40,21 +54,6 @@
 import org.apache.pulsar.io.core.SourceContext;
 import org.slf4j.Logger;
 
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.LinkedList;
-import java.util.Map;
-import java.util.Optional;
-import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.TimeUnit;
-
-import static 

[GitHub] asfgit commented on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

asfgit commented on issue #2200: support subscription-type to be passed in 
sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406437762
 
 
   FAILURE

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie opened a new issue #2207: Add example yaml files for pulsar connectors

[GitBox]

sijie opened a new issue #2207: Add example yaml files for pulsar connectors
URL: https://github.com/apache/incubator-pulsar/issues/2207
 
 
   *Problem*
   
   We don't have any example yaml files for pulsar connectors. We need those 
examples to simplify users on configuring connnectors


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

svn commit: r28239 - in /dev/incubator/pulsar: pulsar-2.1.0-incubating-candidate-4/ pulsar-2.1.0-incubating-candidate-5/ pulsar-2.1.0-incubating-candidate-5/DEB/ pulsar-2.1.0-incubating-candidate-5/RP

[sijie]

Author: sijie
Date: Thu Jul 19 22:36:09 2018
New Revision: 28239

Log:
Staging artifacts and signature for Pulsar release 2.1.0-incubating

Added:
dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/
dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/
dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/Packages.gz   
(with props)
dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/Packages.gz.asc

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/Packages.gz.sha1

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/Packages.gz.sha512

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/apache-pulsar-client-dev.deb
   (with props)

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/apache-pulsar-client-dev.deb.asc

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/apache-pulsar-client-dev.deb.sha1

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/apache-pulsar-client-dev.deb.sha512

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/apache-pulsar-client.deb
   (with props)

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/apache-pulsar-client.deb.asc

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/apache-pulsar-client.deb.sha1

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/DEB/apache-pulsar-client.deb.sha512
dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-2.1.0-1_incubating.x86_64.rpm
   (with props)

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-2.1.0-1_incubating.x86_64.rpm.asc

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-2.1.0-1_incubating.x86_64.rpm.sha1

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-2.1.0-1_incubating.x86_64.rpm.sha512

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-debuginfo-2.1.0-1_incubating.x86_64.rpm
   (with props)

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-debuginfo-2.1.0-1_incubating.x86_64.rpm.asc

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-debuginfo-2.1.0-1_incubating.x86_64.rpm.sha1

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-debuginfo-2.1.0-1_incubating.x86_64.rpm.sha512

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm
   (with props)

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm.asc

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm.sha1

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm.sha512
dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/028e24a11d83290326784618ab7ee1243fa1cd81bba1aefc64a1712c0ec6277f-filelists.xml.gz
   (with props)

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/028e24a11d83290326784618ab7ee1243fa1cd81bba1aefc64a1712c0ec6277f-filelists.xml.gz.asc

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/028e24a11d83290326784618ab7ee1243fa1cd81bba1aefc64a1712c0ec6277f-filelists.xml.gz.sha1

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/028e24a11d83290326784618ab7ee1243fa1cd81bba1aefc64a1712c0ec6277f-filelists.xml.gz.sha512

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/10234562394b36d1476299b5e3071836219f03570f151f37f62a97a9c8d73520-primary.sqlite.bz2
   (with props)

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/10234562394b36d1476299b5e3071836219f03570f151f37f62a97a9c8d73520-primary.sqlite.bz2.asc

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/10234562394b36d1476299b5e3071836219f03570f151f37f62a97a9c8d73520-primary.sqlite.bz2.sha1

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/10234562394b36d1476299b5e3071836219f03570f151f37f62a97a9c8d73520-primary.sqlite.bz2.sha512

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/390cd5e4270286727fcc792638fc7dc815b22413336ebb787c742e34b4e9e57d-other.sqlite.bz2
   (with props)

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/390cd5e4270286727fcc792638fc7dc815b22413336ebb787c742e34b4e9e57d-other.sqlite.bz2.asc

dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/RPMS/repodata/390cd5e4270286727fcc792638fc7dc815b22413336ebb787c742e34b4e9e57d-other.sqlite.bz2.sha1

[incubator-pulsar.wiki] branch master updated: Updated Release Candidate Validation (markdown)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 9813603  Updated Release Candidate Validation (markdown)
9813603 is described below

commit 98136032ab83bfb1eabe84acca66a6ec520d915f
Author: Sijie Guo 
AuthorDate: Thu Jul 19 15:30:58 2018 -0700

Updated Release Candidate Validation (markdown)
---
 Release-Candidate-Validation.md | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index ed7100a..033e0cd 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -3,6 +3,10 @@
   * [Validate Pub/Sub and Java Functions](#validate-pubsub-and-java-functions)
   * [Validate Connectors](#validate-connectors)
   * [Validate Stateful Functions](#validate-stateful-functions)
+- [Validate RPM and DEB packages](#validate-rpm-and-deb-packages)
+  * [Prepare](#prepare)
+  * [Validate RPM](#validate-rpm)
+  * [Validate DEB](#validate-deb)
 
 Here are some instructions for reviewing and validating a release candidate.
 
@@ -518,7 +522,7 @@ value = 10
 value = 20
 ```
 
-### Validate RPM and DEP packages
+### Validate RPM and DEB packages
 
  Prepare
 

[incubator-pulsar.wiki] branch master updated: Updated Release Candidate Validation (markdown)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 041089e  Updated Release Candidate Validation (markdown)
041089e is described below

commit 041089e60abc7a815f883157da07f1d2f2e451e6
Author: Sijie Guo 
AuthorDate: Thu Jul 19 15:29:33 2018 -0700

Updated Release Candidate Validation (markdown)
---
 Release-Candidate-Validation.md | 27 ++-
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index c42d00f..ed7100a 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -537,7 +537,6 @@ in centos container, go to directory `/pulsar`
 ```shell
 [root@e6c29e2b70a9 /]# cd pulsar/
 [root@e6c29e2b70a9 pulsar]# ls
-apache-pulsar-2.1.0-incubating  apache-pulsar-client-dev.deb  
apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm
 ```
 
 2. Install RPM in the centos container
@@ -575,30 +574,26 @@ producer  producer.go
 # ./producer
 ```
 
- Validate RPM
+ Validate DEB
 
-1. Start a centos docker to install RPM
+1. Start an ubuntu docker to install DEB
 
 ```shell
-$ docker run -it --rm -v `pwd`:/pulsar centos:7
+$ docker run -it --rm -v `pwd`:/pulsar ubuntu:16.04
 ```
 
-in centos container, go to directory `/pulsar`
+in ubuntu container, go to directory `/pulsar`
 
 ```shell
 [root@e6c29e2b70a9 /]# cd pulsar/
 [root@e6c29e2b70a9 pulsar]# ls
-apache-pulsar-2.1.0-incubating  apache-pulsar-client-dev.deb  
apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm
 ```
 
 2. Install RPM in the centos container
 
 ```shell
-# rpm -ivh apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm 
apache-pulsar-client-2.1.0-1_incubating.x86_64.rpm
-Preparing...  # [100%]
-Updating / installing...
-   1:apache-pulsar-client-2.1.0-1_incu# [ 50%]
-   2:apache-pulsar-client-devel-2.1.0-# [100%]
+# apt install ./apache-pulsar-client.deb
+# apt install ./apache-pulsar-client-dev.deb
 ```
 
 3. Build the go client to validate the RPM package.
@@ -606,7 +601,14 @@ Updating / installing...
 a) install go
 
 ```shell
-# yum install -y go git
+# apt-get update
+# apt-get -y upgrade
+# apt-get install -y git curl gcc
+# curl -O https://storage.googleapis.com/golang/go1.9.1.linux-amd64.tar.gz
+# tar -xvf go1.9.1.linux-amd64.tar.gz
+# mv go /usr/local
+# export GOROOT=/usr/local/go
+# export PATH=$PATH:$GOROOT/bin
 ```
 
 b) get pulsar go client
@@ -622,6 +624,5 @@ c) build an example
 # go build .
 # ls
 producer  producer.go
-# ldconfig
 # ./producer
 ```
\ No newline at end of file

[incubator-pulsar.wiki] branch master updated: Updated Release Candidate Validation (markdown)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new f51bf79  Updated Release Candidate Validation (markdown)
f51bf79 is described below

commit f51bf79e3c684166cf88242285210ae5b08a46a6
Author: Sijie Guo 
AuthorDate: Thu Jul 19 15:10:42 2018 -0700

Updated Release Candidate Validation (markdown)
---
 Release-Candidate-Validation.md | 108 
 1 file changed, 108 insertions(+)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index 4ab0238..c42d00f 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -516,4 +516,112 @@ The result in the terminal open at step 3 is updated to 
`20`.
 ```shell
 value = 10
 value = 20
+```
+
+### Validate RPM and DEP packages
+
+ Prepare
+
+Download the RPM and DEP packages to a directory. Let's name this directory as 
`pulsar_validation`. We will mount this directory to a docker container so that 
we can validate RPM packages.
+
+ Validate RPM
+
+1. Start a centos docker to install RPM
+
+```shell
+$ docker run -it --rm -v `pwd`:/pulsar centos:7
+```
+
+in centos container, go to directory `/pulsar`
+
+```shell
+[root@e6c29e2b70a9 /]# cd pulsar/
+[root@e6c29e2b70a9 pulsar]# ls
+apache-pulsar-2.1.0-incubating  apache-pulsar-client-dev.deb  
apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm
+```
+
+2. Install RPM in the centos container
+
+```shell
+# rpm -ivh apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm 
apache-pulsar-client-2.1.0-1_incubating.x86_64.rpm
+Preparing...  # [100%]
+Updating / installing...
+   1:apache-pulsar-client-2.1.0-1_incu# [ 50%]
+   2:apache-pulsar-client-devel-2.1.0-# [100%]
+```
+
+3. Build the go client to validate the RPM package.
+
+a) install go
+
+```shell
+# yum install -y go git
+```
+
+b) get pulsar go client
+
+```shell
+# go get -u github.com/apache/incubator-pulsar/pulsar-client-go/pulsar
+```
+
+c) build an example
+
+```shell
+# cd 
~/go/src/github.com/apache/incubator-pulsar/pulsar-client-go/examples/producer/
+# go build .
+# ls
+producer  producer.go
+# ldconfig
+# ./producer
+```
+
+ Validate RPM
+
+1. Start a centos docker to install RPM
+
+```shell
+$ docker run -it --rm -v `pwd`:/pulsar centos:7
+```
+
+in centos container, go to directory `/pulsar`
+
+```shell
+[root@e6c29e2b70a9 /]# cd pulsar/
+[root@e6c29e2b70a9 pulsar]# ls
+apache-pulsar-2.1.0-incubating  apache-pulsar-client-dev.deb  
apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm
+```
+
+2. Install RPM in the centos container
+
+```shell
+# rpm -ivh apache-pulsar-client-devel-2.1.0-1_incubating.x86_64.rpm 
apache-pulsar-client-2.1.0-1_incubating.x86_64.rpm
+Preparing...  # [100%]
+Updating / installing...
+   1:apache-pulsar-client-2.1.0-1_incu# [ 50%]
+   2:apache-pulsar-client-devel-2.1.0-# [100%]
+```
+
+3. Build the go client to validate the RPM package.
+
+a) install go
+
+```shell
+# yum install -y go git
+```
+
+b) get pulsar go client
+
+```shell
+# go get -u github.com/apache/incubator-pulsar/pulsar-client-go/pulsar
+```
+
+c) build an example
+
+```shell
+# cd 
~/go/src/github.com/apache/incubator-pulsar/pulsar-client-go/examples/producer/
+# go build .
+# ls
+producer  producer.go
+# ldconfig
+# ./producer
 ```
\ No newline at end of file

[GitHub] merlimat commented on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

merlimat commented on issue #2200: support subscription-type to be passed in 
sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406426023
 
 
   I don't think this should be a blocker for 2.1. We are already very late and 
left out other fixes, otherwise we keep finding issues or improvements to do 
and we never actually cut the release :). 
   
   To retain ordering one can always use effectively once mode on the function. 
   
   In general, I would rather have a parameter called "retain ordering " in the 
function configuration, which might be easier to understand than 
shaheed/failover 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia commented on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

rdhabalia commented on issue #2200: support subscription-type to be passed in 
sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406422509
 
 
   > I think we used to have subscription type but we removed it sometime ago.
   
   umm.. then how can we maintain the ordering without it ?
   
   > regarding releases, can we include this as 2.1.1 release, not to delay 
2.1.0 release?
   
   my only concern was that user might not be able to use sink-connector with 
2.1.0 if they need message ordering guarantee, else I am fine. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia commented on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

rdhabalia commented on issue #2200: support subscription-type to be passed in 
sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406420204
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

sijie commented on issue #2200: support subscription-type to be passed in 
sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406414109
 
 
   @rdhabalia I think we used to have subscription type but we removed it 
sometime ago. @jerrypeng and @srkukarni have more context.
   
   regarding releases, can we include this as 2.1.1 release, not to delay 2.1.0 
release? thoughts @merlimat ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] annotated tag v2.1.0-incubating-candidate-5 updated (75f79b7 -> d194561)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a change to annotated tag v2.1.0-incubating-candidate-5
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git.


*** WARNING: tag v2.1.0-incubating-candidate-5 was modified! ***

from 75f79b7  (commit)
  to d194561  (tag)
 tagging 75f79b77f55fbad351cb517e7f3b8ba4ab2dc177 (commit)
  by Sijie Guo
  on Thu Jul 19 14:00:45 2018 -0700

- Log -
Release v2.1.0-incubating-candidate-5
-BEGIN PGP SIGNATURE-

iQJFBAABCgAvFiEE2k3oIDhnJWi8g07gWz+9tP10QCwFAltQ+/0RHHNpamllQGFw
YWNoZS5vcmcACgkQWz+9tP10QCxY4Q/+Ne/IndleakOf4bgBKix2y1NKgsSMRkBy
+78MsJ715JwG6eMBBgtRIEimaSBdcmukI2kcpJ2vdyX9jvthPQT1MNF+QFIMmyoX
8TpZFDZdcwN4seaukMHhcyM8aMDswc5/0Ro9qCVs7/vMxVoZJoOVMjJmXst4LM3r
079W8BcfgRunTwfif/20wJi+4CqBWZUM1PoWY/GpbHgrNlU1sSob7NOWRkSaOxUr
VkajUfycOwPydX0IKEntMAzVrW4lR+3xCrUkos/aXuXbZlsUDmATmHrZ+d1Sg0Pt
ZN9QzhR1MX5vUcETrQw3pgfb32OjuDsusTklbkMxY0lajjcrorChf8vu2ZBVBra3
IWtiCz49zkNFoN1OKUYmzupqgkoWmCFW45JyNKA3SKRivqwi5jdTLRhJ3XCrO5Aw
PUQ9iMT/fkTpLD1u5yucfGgbHpNOH48cGxXE3iS6QfwoTzAGohLrHXsYuuj8mdWg
R9gahA4V5m+0nETj3VyHnUHNc9kPSAtdTDll19PVGNGa4QGDIRE2HvRKZTlZkoO9
GFBcksBjNLoE7TDZHtRjdpLAnKLqjXeIdQpsTnUEowjuMNMi6lTuQbBOvXSXqEaJ
hsOC3rQsUH77IhexIdxw/L5+VwkFylfpw4o6jgluQ6RSWp9zVhXhVVyQLSKE6szO
fIsR++dV2Fw=
=49P5
-END PGP SIGNATURE-
---


No new revisions were added by this update.

Summary of changes:

[incubator-pulsar] 01/02: Fix License Issue

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git

commit e1c7df8c43e69473aa8a4108232c0c474134a28e
Author: Sijie Guo 
AuthorDate: Thu Jul 19 13:57:13 2018 -0700

Fix License Issue

 ### Motivation

Some libaries are not bundled with server distribution anymore.

 ### Changes

Update the LICENSE.bin.txt to reflect the libraries we have in server 
distribution.
---
 distribution/server/src/assemble/LICENSE.bin.txt | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/distribution/server/src/assemble/LICENSE.bin.txt 
b/distribution/server/src/assemble/LICENSE.bin.txt
index 33b9748..d813041 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -349,7 +349,7 @@ The Apache Software License, Version 2.0
 - commons-lang-commons-lang-2.6.jar
 - commons-logging-commons-logging-1.1.1.jar
 - org.apache.commons-commons-collections4-4.1.jar
-- org.apache.commons-commons-compress-1.8.1.jar
+- org.apache.commons-commons-compress-1.15.jar
 - org.apache.commons-commons-lang3-3.4.jar
  * Netty
 - io.netty-netty-3.10.1.Final.jar
@@ -448,7 +448,6 @@ The Apache Software License, Version 2.0
 - net.jodah-typetools-0.5.0.jar
   * Apache Avro
 - org.apache.avro-avro-1.8.2.jar
-- org.apache.avro-avro-protobuf-1.8.2.jar
   * Apache Curator
 - org.apache.curator-curator-client-4.0.1.jar
 - org.apache.curator-curator-framework-4.0.1.jar
@@ -459,8 +458,8 @@ The Apache Software License, Version 2.0
 - org.inferred-freebuilder-1.14.9.jar
   * Snappy Java
 - org.xerial.snappy-snappy-java-1.1.1.3.jar
-  * Flatbuffers Java
-- com.google.flatbuffers-flatbuffers-java-1.9.0.jar
+  * Objenesis
+- org.objenesis-objenesis-2.1.jar
 
 
 BSD 3-clause "New" or "Revised" License

[incubator-pulsar] 02/02: Exclude helm files from Apache Rat Check.

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git

commit 75f79b77f55fbad351cb517e7f3b8ba4ab2dc177
Author: Sijie Guo 
AuthorDate: Thu Jul 19 13:58:19 2018 -0700

Exclude helm files from Apache Rat Check.

*Motivation*

Some of the helm files (e.g. helm ignore file and template file) doesn't 
have ASF license header.
It is tricky to add license headers for them. so exclude them from rat 
check.

*Changes*

Exclude two files from Apache Rat Check
---
 pom.xml | 4 
 1 file changed, 4 insertions(+)

diff --git a/pom.xml b/pom.xml
index 40c9a10..0a2fae5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1113,6 +1113,10 @@ flexible messaging model and an intuitive client 
API.
 
 
 conf/schema_example.conf
+
+
+**/.helmignore
+**/_helpers.tpl
   
 
   

[incubator-pulsar] branch branch-2.1 updated (a3082c1 -> 75f79b7)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git.


from a3082c1  Release 2.1.0-incubating
 new e1c7df8  Fix License Issue
 new 75f79b7  Exclude helm files from Apache Rat Check.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 distribution/server/src/assemble/LICENSE.bin.txt | 7 +++
 pom.xml  | 4 
 2 files changed, 7 insertions(+), 4 deletions(-)

[incubator-pulsar.wiki] branch master updated: Updated Release process (markdown)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 8265b8d  Updated Release process (markdown)
8265b8d is described below

commit 8265b8d74e1162349e4bfff394f4a4dfa359b146
Author: Sijie Guo 
AuthorDate: Thu Jul 19 13:56:46 2018 -0700

Updated Release process (markdown)
---
 Release-process.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Release-process.md b/Release-process.md
index 351c253..c865e85 100644
--- a/Release-process.md
+++ b/Release-process.md
@@ -90,7 +90,7 @@ Inspect the artifacts:
  * Check that the `LICENSE` and `NOTICE` files cover all included jars 
(especially for the -bin package)
- Use script to cross-validate `LICENSE` file with included jars: 
   ```
-  src/check-binary-license 
all/target/apache-pulsar-1.X.0-incubating-bin.tar.gz
+  src/check-binary-license 
distribution/server/target/apache-pulsar-1.x.0-incubating-bin.tar.gz
   ```
  * Run Apache RAT to verify the license headers in the `src` package:
  ```shell
@@ -101,6 +101,8 @@ Inspect the artifacts:
  bin/pulsar standalone
  ```
 
+* Use instructions in 
[Release-Candidate-Validation](Release-Candidate-Validation) to do some sanity 
checks on the produced binary distributions.
+
 # 3.1. Build RPM and DEB packages
 
 ```shell

[GitHub] cckellogg opened a new pull request #2206: Site2

[GitBox]

cckellogg opened a new pull request #2206: Site2
URL: https://github.com/apache/incubator-pulsar/pull/2206
 
 
   ### Motivation
   
   Improve the documentation and usability of the pulsar website. This moves 
the website and documentation to a new framework (https://docusaurus.io/)  
which will make it easier to maintain going forward.
   
   ### Modifications
   
   A new version of the website in site2 directory. Also updates the pulsar 
build docker to add the new website build dependencies.
   
   ### Result
   
   A more usable website and documentation.
   
   A preview of the site can be seen here: 
https://cckellogg.github.io/incubator-pulsar
   *All the links and images might not work on this site since it's a test only 
site*
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar.wiki] branch master updated: add toc

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new c5089d4  add toc
c5089d4 is described below

commit c5089d4bcdd90c57880df9ad72ecf9fd47c4ceb1
Author: Sijie Guo 
AuthorDate: Thu Jul 19 13:29:08 2018 -0700

add toc
---
 Release-Candidate-Validation.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index 08491f8..4ab0238 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -2,6 +2,7 @@
   * [Download Binary Distributions](#download-the-binary-distributions)
   * [Validate Pub/Sub and Java Functions](#validate-pubsub-and-java-functions)
   * [Validate Connectors](#validate-connectors)
+  * [Validate Stateful Functions](#validate-stateful-functions)
 
 Here are some instructions for reviewing and validating a release candidate.
 
@@ -419,7 +420,7 @@ $ bin/pulsar-admin sink delete --tenant public --namespace 
default --name cassan
 "Deleted successfully"
 ```
 
- Validate Counter Functions
+ Validate Stateful Functions
 
 Since Pulsar 2.1, Pulsar enables bookkeeper table service for stateful pulsar 
functions (as a developer preview).
 

[incubator-pulsar.wiki] branch master updated: instructions for word count

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 1929fda  instructions for word count
1929fda is described below

commit 1929fdac03dfaf711e79ff9be669949a552a3856
Author: Sijie Guo 
AuthorDate: Thu Jul 19 13:28:13 2018 -0700

instructions for word count
---
 Release-Candidate-Validation.md | 98 +
 1 file changed, 98 insertions(+)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index 9492b51..08491f8 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -418,3 +418,101 @@ cqlsh:pulsar_test_keyspace> select * from 
pulsar_test_table;
 $ bin/pulsar-admin sink delete --tenant public --namespace default --name 
cassandra-test-sink
 "Deleted successfully"
 ```
+
+ Validate Counter Functions
+
+Since Pulsar 2.1, Pulsar enables bookkeeper table service for stateful pulsar 
functions (as a developer preview).
+
+Here are the instructions to validate counter functions:
+
+1. Create a wordcount function
+
+```shell
+$ bin/pulsar-admin functions create --functionConfigFile 
examples/example-function-config.yaml --jar examples/api-examples.jar --name 
word_count --className 
org.apache.pulsar.functions.api.examples.WordCountFunction --inputs 
test_wordcount_src --output test_wordcount_dest
+"Created successfully"
+```
+
+2. Get function info and status
+
+```shell
+$ bin/pulsar-admin functions get --tenant test --namespace test-namespace 
--name word_count
+{
+  "tenant": "test",
+  "namespace": "test-namespace",
+  "name": "word_count",
+  "className": "org.apache.pulsar.functions.api.examples.WordCountFunction",
+  "userConfig": "{\"PublishTopic\":\"test_result\"}",
+  "autoAck": true,
+  "parallelism": 1,
+  "source": {
+"topicsToSerDeClassName": {
+  "test_wordcount_src": ""
+},
+"typeClassName": "java.lang.String"
+  },
+  "sink": {
+"topic": "test_wordcount_dest",
+"typeClassName": "java.lang.Void"
+  },
+  "resources": {}
+}
+```
+
+```shell
+$ bin/pulsar-admin functions getstatus --tenant test --namespace 
test-namespace --name word_count
+{
+  "functionStatusList": [
+{
+  "running": true,
+  "instanceId": "0",
+  "metrics": {
+"metrics": {
+  "__total_processed__": {},
+  "__total_successfully_processed__": {},
+  "__total_system_exceptions__": {},
+  "__total_user_exceptions__": {},
+  "__total_serialization_exceptions__": {},
+  "__avg_latency_ms__": {}
+}
+  },
+  "workerId": "c-standalone-fw-localhost-6750"
+}
+  ]
+}
+```
+
+3. Query the state table for the function: watching on a key called "hello"
+
+```shell
+$ bin/pulsar-admin functions querystate --tenant test --namespace 
test-namespace --name word_count -u bk://localhost:4181 -k hello -w
+key 'hello' doesn't exist.
+key 'hello' doesn't exist.
+key 'hello' doesn't exist
+```
+
+4. Produce the messages to source topic `test_wordcount_src`.
+
+Produce 10 messages "hello" to topic `test_wordcount_src`. The value of 
"hello" should be updated to 10.
+
+```shell
+$ bin/pulsar-client produce -m "hello" -n 10 test_wordcount_src
+```
+
+Checkout the result in the terminal open at step 3.
+
+```shell
+value = 10
+```
+
+Now produce another 10 messages "hello". You will see the result is updated to 
20.
+
+```shell
+$ bin/pulsar-client produce -m "hello" -n 10 test_wordcount_src
+```
+
+The result in the terminal open at step 3 is updated to `20`.
+
+```shell
+value = 10
+value = 20
+```
\ No newline at end of file

[incubator-pulsar.wiki] branch master updated: add toc

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new e1f48ed  add toc
e1f48ed is described below

commit e1f48ed12188d7ce2e8180762945724e6ea844a9
Author: Sijie Guo 
AuthorDate: Thu Jul 19 13:17:09 2018 -0700

add toc
---
 Release-Candidate-Validation.md | 5 +
 1 file changed, 5 insertions(+)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index 39fd899..f9a3c6b 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -1,3 +1,8 @@
+- [Validate Binary Distribution](#validate-the-binary-distribution)
+  * [Download Binary Distributions](#download-the-binary-distributions)
+  * [Validate Pub/Sub and Java Functions](#validate-pub-sub-and-java-functions)
+  * [Validate Connectors](#validate-connectors)
+
 Here are some instructions for reviewing and validating a release candidate.
 
 ### Validate the binary distribution

[incubator-pulsar.wiki] branch master updated: fix toc

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new b04be4a  fix toc
b04be4a is described below

commit b04be4ae6a3a03863cb2e2d00f86fd213c4291b4
Author: Sijie Guo 
AuthorDate: Thu Jul 19 13:17:45 2018 -0700

fix toc
---
 Release-Candidate-Validation.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index f9a3c6b..9492b51 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -1,6 +1,6 @@
 - [Validate Binary Distribution](#validate-the-binary-distribution)
   * [Download Binary Distributions](#download-the-binary-distributions)
-  * [Validate Pub/Sub and Java Functions](#validate-pub-sub-and-java-functions)
+  * [Validate Pub/Sub and Java Functions](#validate-pubsub-and-java-functions)
   * [Validate Connectors](#validate-connectors)
 
 Here are some instructions for reviewing and validating a release candidate.

[incubator-pulsar.wiki] branch master updated: validate connectors

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new a11329a  validate connectors
a11329a is described below

commit a11329a56f31739322b39b7feb359772ef15ec4f
Author: Sijie Guo 
AuthorDate: Thu Jul 19 13:13:48 2018 -0700

validate connectors
---
 Release-Candidate-Validation.md | 140 
 1 file changed, 140 insertions(+)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index db9bb91..39fd899 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -273,3 +273,143 @@ cqlsh> USE pulsar_test_keyspace;
 cqlsh:pulsar_test_keyspace> CREATE TABLE pulsar_test_table (key text PRIMARY 
KEY, col text);
 
 ```
+
+3. Prepare a cassandra sink yaml file and put it under examples directory as 
`cassandra-sink.yml`.
+
+```shell
+$ cat examples/cassandra-sink.yml
+configs:
+roots: "localhost:9042"
+keyspace: "pulsar_test_keyspace"
+columnFamily: "pulsar_test_table"
+keyname: "key"
+columnName: "col"
+```
+
+4. Submit a cassandra sink
+
+```shell
+$ bin/pulsar-admin sink create --tenant public --namespace default --name 
cassandra-test-sink --sink-type cassandra --sinkConfigFile 
examples/cassandra-sink.yml --inputs test_cassandra
+"Created successfully"
+```
+
+```shell
+// get the sink info
+$ bin/pulsar-admin functions get --tenant public --namespace default --name 
cassandra-test-sink
+{
+  "tenant": "public",
+  "namespace": "default",
+  "name": "cassandra-test-sink",
+  "className": "org.apache.pulsar.functions.api.utils.IdentityFunction",
+  "autoAck": true,
+  "parallelism": 1,
+  "source": {
+"topicsToSerDeClassName": {
+  "test_cassandra": ""
+}
+  },
+  "sink": {
+"configs": 
"{\"roots\":\"cassandra\",\"keyspace\":\"pulsar_test_keyspace\",\"columnFamily\":\"pulsar_test_table\",\"keyname\":\"key\",\"columnName\":\"col\"}",
+"builtin": "cassandra"
+  },
+  "resources": {}
+}
+```
+
+```shell
+// get the running status
+$ bin/pulsar-admin functions getstatus --tenant public --namespace default 
--name cassandra-test-sink
+{
+  "functionStatusList": [
+{
+  "running": true,
+  "instanceId": "0",
+  "metrics": {
+"metrics": {
+  "__total_processed__": {},
+  "__total_successfully_processed__": {},
+  "__total_system_exceptions__": {},
+  "__total_user_exceptions__": {},
+  "__total_serialization_exceptions__": {},
+  "__avg_latency_ms__": {}
+}
+  },
+  "workerId": "c-standalone-fw-localhost-6750"
+}
+  ]
+}
+```
+
+5. Produce messages to the source topic
+```shell
+$ for i in {0..10}; do bin/pulsar-client produce -m "key-$i" -n 1 
test_cassandra; done
+```
+
+6. Check the sink status. It should show 11 messages are processed.
+
+```shell
+$ bin/pulsar-admin functions getstatus --tenant public --namespace default 
--name cassandra-test-sink
+{
+  "functionStatusList": [
+{
+  "running": true,
+  "numProcessed": "11",
+  "numSuccessfullyProcessed": "11",
+  "lastInvocationTime": "1532031040117",
+  "instanceId": "0",
+  "metrics": {
+"metrics": {
+  "__total_processed__": {
+"count": 5.0,
+"sum": 5.0,
+"max": 5.0
+  },
+  "__total_successfully_processed__": {
+"count": 5.0,
+"sum": 5.0,
+"max": 5.0
+  },
+  "__total_system_exceptions__": {},
+  "__total_user_exceptions__": {},
+  "__total_serialization_exceptions__": {},
+  "__avg_latency_ms__": {}
+}
+  },
+  "workerId": "c-standalone-fw-localhost-6750"
+}
+  ]
+}
+```
+
+7. Check results in cassandra
+
+```shell
+$ docker exec -ti cassandra cqlsh localhost
+Connected to Test Cluster at localhost:9042.
+[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
+Use HELP for help.
+cqlsh> use pulsar_test_keyspace;
+cqlsh:pulsar_test_keyspace> select * from pulsar_test_table;
+
+ key| col
++
+  key-5 |  key-5
+  key-0 |  key-0
+  key-9 |  key-9
+  key-2 |  key-2
+  key-1 |  key-1
+  key-3 |  key-3
+  key-6 |  key-6
+  key-7 |  key-7
+  key-4 |  key-4
+  key-8 |  key-8
+ key-10 | key-10
+
+(11 rows)
+```
+8. Delete the Sink
+
+```shell
+$ bin/pulsar-admin sink delete --tenant public --namespace default --name 
cassandra-test-sink
+"Deleted successfully"
+```

[incubator-pulsar.wiki] branch master updated: startup cassandra cluster

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new cf2661b  startup cassandra cluster
cf2661b is described below

commit cf2661b50d2017282cb0ea0c03a4405d35507e3d
Author: Sijie Guo 
AuthorDate: Thu Jul 19 12:47:14 2018 -0700

startup cassandra cluster
---
 Release-Candidate-Validation.md | 59 -
 1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index 6a1aa94..db9bb91 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -215,4 +215,61 @@ test-messages-Thu Jul 19 11:59:15 PDT 2018!
 test-messages-Thu Jul 19 11:59:15 PDT 2018!
 - got message -
 test-messages-Thu Jul 19 11:59:15 PDT 2018!
-```
\ No newline at end of file
+```
+
+ Validate Connectors
+
+> Make sure you have docker available at your laptop. If you don't have docker 
installed, you can skip this section.
+
+1. Setup a cassandra cluster.
+
+```shell
+$ docker run -d --rm  --name=cassandra -p 9042:9042 cassandra
+```
+
+Make sure the cassandra cluster is up running.
+
+```shell
+// run docker ps to find the docker process for cassandra
+$ docker ps
+```
+
+```shell
+// check if the cassandra is running as expected
+$ docker logs cassandra
+```
+
+```shell
+// check the cluster status
+$ docker exec cassandra nodetool status
+Datacenter: datacenter1
+===
+Status=Up/Down
+|/ State=Normal/Leaving/Joining/Moving
+--  Address Load   Tokens   Owns (effective)  Host ID  
 Rack
+UN  172.17.0.2  103.67 KiB  256  100.0%
af0e4b2f-84e0-4f0b-bb14-bd5f9070ff26  rack1
+```
+
+```shell
+
+```
+
+2. Create keyspace and table
+
+Run cqlsh:
+```shell
+$ docker exec -ti cassandra cqlsh localhost
+Connected to Test Cluster at localhost:9042.
+[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
+Use HELP for help.
+cqlsh>
+```
+
+In the cqlsh, create keyspace `pulsar_test_keyspace` and table 
`pulsar_test_table`.
+
+```shell
+cqlsh> CREATE KEYSPACE pulsar_test_keyspace WITH replication = 
{'class':'SimpleStrategy', 'replication_factor':1}; 
+cqlsh> USE pulsar_test_keyspace;
+cqlsh:pulsar_test_keyspace> CREATE TABLE pulsar_test_table (key text PRIMARY 
KEY, col text);
+
+```

[incubator-pulsar.wiki] branch master updated: Updated PIP 18: Pulsar SQL (markdown)

[jerrypeng]

This is an automated email from the ASF dual-hosted git repository.

jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new a190d49  Updated PIP 18: Pulsar SQL (markdown)
a190d49 is described below

commit a190d49d6128a93533e5acbecd4fb4a1da83f752
Author: Boyang Jerry Peng 
AuthorDate: Thu Jul 19 12:39:36 2018 -0700

Updated PIP 18: Pulsar SQL (markdown)
---
 PIP-18:-Pulsar-SQL.md | 12 +++-
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/PIP-18:-Pulsar-SQL.md b/PIP-18:-Pulsar-SQL.md
index 50333c7..716f64b 100644
--- a/PIP-18:-Pulsar-SQL.md
+++ b/PIP-18:-Pulsar-SQL.md
@@ -9,10 +9,13 @@ N/A
 We are trying to create a method in which users can explore, in a natural 
manner,  the data already stored within Pulsar topics.  We believe the best way 
to accomplish this is to expose SQL interface that allows users to query 
existing data within a Pulsar cluster.  
 
 Just to be absolutely clear,  the SQL we are proposing is for querying data 
already in Pulsar and we are currently not proposing the implementation of any 
sort of SQL on data streams
-Why are we doing this?
 
-Many users are interested for such a feature.  For example, many users store 
large amounts of historical data in Pulsar for various purposes.  Giving them 
to capability to query that that data gives them huge value.  Users will 
typically need to stream the data out of Pulsar and into another platform to do 
any sort of analysis, but with Pulsar SQL, users can just use one platform.
-How are we going to do it?
+
+## Why are we doing this?
+
+Many users are interested in such a feature.  For example, many users store 
large amounts of historical data in Pulsar for various purposes.  Giving them 
to capability to query that that data gives them huge value.  Users will 
typically need to stream the data out of Pulsar and into another platform to do 
any sort of analysis, but with Pulsar SQL, users can just use one platform.
+
+## How are we going to do it?
 
 With the implementation of a schema registry in Pulsar, data can be structured 
so that it can be easily mapped to tables that can be queried by SQL. We plan 
on using Presto (https://prestodb.io/) as the backbone of Pulsar SQL.  A 
connector can be implemented using the Presto connector SPI that allows presto 
to ingest data from Pulsar and to be queried using Presto’s existing SQL 
framework.
 
@@ -21,7 +24,6 @@ The schema registry will be used to generate the structure of 
tables that will b
 Thus, Pulsar will be queried for metadata concerning topics and schemas and 
from that metadata, we will go directly to the bookies to load and deserialize 
the data.
 
 
-
 ## Goals
 
 * Allow users to submit SQL queries using a Pulsar CLI
@@ -92,4 +94,4 @@ Let’s break the implementation into multiple phases:
 6. Performance testing and optimizing
 
 
-More to come...
\ No newline at end of file
+More to come...

[incubator-pulsar.wiki] branch master updated: Updated PIP 18: Pulsar SQL (markdown)

[jerrypeng]

This is an automated email from the ASF dual-hosted git repository.

jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 4d73a26  Updated PIP 18: Pulsar SQL (markdown)
4d73a26 is described below

commit 4d73a26e0dfc3970cd84efa0616008d8455b275a
Author: Boyang Jerry Peng 
AuthorDate: Thu Jul 19 12:38:01 2018 -0700

Updated PIP 18: Pulsar SQL (markdown)
---
 PIP-18:-Pulsar-SQL.md | 90 +++
 1 file changed, 90 insertions(+)

diff --git a/PIP-18:-Pulsar-SQL.md b/PIP-18:-Pulsar-SQL.md
index 53691a3..50333c7 100644
--- a/PIP-18:-Pulsar-SQL.md
+++ b/PIP-18:-Pulsar-SQL.md
@@ -3,3 +3,93 @@
 * **Mailing List discussion**: 
 N/A
 * **Prototype**: https://github.com/jerrypeng/presto/tree/pulsar_connector
+
+## What are we trying to do?
+
+We are trying to create a method in which users can explore, in a natural 
manner,  the data already stored within Pulsar topics.  We believe the best way 
to accomplish this is to expose SQL interface that allows users to query 
existing data within a Pulsar cluster.  
+
+Just to be absolutely clear,  the SQL we are proposing is for querying data 
already in Pulsar and we are currently not proposing the implementation of any 
sort of SQL on data streams
+Why are we doing this?
+
+Many users are interested for such a feature.  For example, many users store 
large amounts of historical data in Pulsar for various purposes.  Giving them 
to capability to query that that data gives them huge value.  Users will 
typically need to stream the data out of Pulsar and into another platform to do 
any sort of analysis, but with Pulsar SQL, users can just use one platform.
+How are we going to do it?
+
+With the implementation of a schema registry in Pulsar, data can be structured 
so that it can be easily mapped to tables that can be queried by SQL. We plan 
on using Presto (https://prestodb.io/) as the backbone of Pulsar SQL.  A 
connector can be implemented using the Presto connector SPI that allows presto 
to ingest data from Pulsar and to be queried using Presto’s existing SQL 
framework.
+
+The schema registry will be used to generate the structure of tables that will 
be used in Presto.  Presto workers will load data directly from bookies through 
a read-only managed-ledger interface, so that we can have a many to many 
throughput and avoid impacting brokers with read activity.
+
+Thus, Pulsar will be queried for metadata concerning topics and schemas and 
from that metadata, we will go directly to the bookies to load and deserialize 
the data.
+
+
+
+## Goals
+
+* Allow users to submit SQL queries using a Pulsar CLI
+* Throttling
+   * Maintain SLAs between reads for SQL and consumer reads
+* Resource Isolation
+   * Read priorities at BookKeeper level - Bookies should be able to 
support different classes of read requests and prioritize regular consumers vs 
read requests coming from SQL queries
+   * Queries from users or groups should not overly impact other queries 
from other users or groups
+* We would like to target two types of deployments:
+   1. Embedded with pulsar cluster
+   * Run Presto coordinator and workers as part of the existing 
function worker service. 
+   2. Allow existing presto clusters to be used to query a Pulsar cluster
+
+## Implementation
+
+### Presto-Pulsar connector
+
+Presto has a SPI system to add connectors. We need to create a connector able 
to perform the following tasks: 
+* Discover schema for a particular topic
+   * This will be done through Pulsar Schema REST API
+* Discover the amount of entries stored in a topic and break down the entries 
across multiple workers
+* On each worker we need to fetch the entries from Bookies
+   * Use managed ledger in read-only mode
+   * Each worker positions itself on a particular entry and read a 
determined number of entries
+   * Parse Pulsar message metadata and extract messages from the 
BookKeeper entries
+   * Deserialize entries, based on schema definition, and feed the objects 
to Presto
+   
+
+### Presto deployment
+
+Presto is composed of a coordination and multiple worker nodes 
(https://prestodb.io/overview.html). We plan to have Presto run in embedded 
mode with regular Pulsar scripts for uniform operations. 
+
+One of the nodes will be elected as coordinator (in the same way as function 
workers elect a leader already). Since all the requests to Presto coordinator 
are made through HTTP/HTTPS, it would be possible to proxy/redirect requests 
made through the regular Pulsar service URL. 
+
+Regarding worker nodes, one possibility is to co-locate with Pulsar function 
worker nodes, which can in their turn be co-located with brokers, when 
deploying on a small single-tenant cluster.
+
+## Execution Plan
+
+Let’s break the implementation into 

[GitHub] rdhabalia commented on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

rdhabalia commented on issue #2200: support subscription-type to be passed in 
sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406388383
 
 
   @sijie I think we should consider it for 2.1.0 because it will be required 
if someone wants to use sink with ordering guarantee. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar.wiki] branch master updated: Updated PIP 18: Pulsar SQL (markdown)

[jerrypeng]

This is an automated email from the ASF dual-hosted git repository.

jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 35c983f  Updated PIP 18: Pulsar SQL (markdown)
35c983f is described below

commit 35c983fd828ecedbdcec427735b59df6b5b36515
Author: Boyang Jerry Peng 
AuthorDate: Thu Jul 19 12:24:58 2018 -0700

Updated PIP 18: Pulsar SQL (markdown)
---
 PIP-18:-Pulsar-SQL.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/PIP-18:-Pulsar-SQL.md b/PIP-18:-Pulsar-SQL.md
index bee8d4f..53691a3 100644
--- a/PIP-18:-Pulsar-SQL.md
+++ b/PIP-18:-Pulsar-SQL.md
@@ -1,5 +1,5 @@
 * **Status**: In Progress
-* **Authors**: Jerry Peng
+* **Authors**: Jerry Peng, Matteo Merril
 * **Mailing List discussion**: 
 N/A
 * **Prototype**: https://github.com/jerrypeng/presto/tree/pulsar_connector

[incubator-pulsar.wiki] branch master updated: Updated PIP 18 Pulsar SQL (markdown)

[jerrypeng]

This is an automated email from the ASF dual-hosted git repository.

jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new a9793ea  Updated PIP 18 Pulsar SQL (markdown)
a9793ea is described below

commit a9793eafeeeac3680aa700c3403191a892cd6672
Author: Boyang Jerry Peng 
AuthorDate: Thu Jul 19 12:24:11 2018 -0700

Updated PIP 18 Pulsar SQL (markdown)
---
 PIP-18-Pulsar-SQL.md => PIP-18:-Pulsar-SQL.md | 0
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/PIP-18-Pulsar-SQL.md b/PIP-18:-Pulsar-SQL.md
similarity index 100%
rename from PIP-18-Pulsar-SQL.md
rename to PIP-18:-Pulsar-SQL.md

[incubator-pulsar.wiki] branch master updated: add sanity checks

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 07d8336  add sanity checks
07d8336 is described below

commit 07d83363b3ee81fa15fe3b31aebd5b11fc8e
Author: Sijie Guo 
AuthorDate: Thu Jul 19 12:25:02 2018 -0700

add sanity checks
---
 Release-Candidate-Validation.md | 73 +
 1 file changed, 73 insertions(+)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index be734db..6a1aa94 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -36,6 +36,79 @@ pulsar-io-twitter-.nar
 $ bin/pulsar standalone
 ```
 
+when you started the standalone cluster, there are a few things to example:
+
+a) the standalone cluster should be able to locate all the connectors. you 
should be able to see following logging information.
+
+```shell
+Found connector ConnectorDefinition(name=kinesis, description=Kinesis sink 
connector, sourceClass=null, 
sinkClass=org.apache.pulsar.io.kinesis.KinesisSink) from 
/Users/sijie/tmp/apache-pulsar-2.1.0-incubating/./connectors/pulsar-io-kinesis-2.1.0-incubating.nar
+...
+Found connector ConnectorDefinition(name=cassandra, description=Writes data 
into Cassandra, sourceClass=null, 
sinkClass=org.apache.pulsar.io.cassandra.CassandraStringSink) from 
/Users/sijie/tmp/apache-pulsar-2.1.0-incubating/./connectors/pulsar-io-cassandra-2.1.0-incubating.nar
+...
+Found connector ConnectorDefinition(name=aerospike, description=Aerospike 
database sink, sourceClass=null, 
sinkClass=org.apache.pulsar.io.aerospike.AerospikeStringSink) from 
/Users/sijie/tmp/apache-pulsar-2.1.0-incubating/./connectors/pulsar-io-aerospike-2.1.0-incubating.nar
+```
+
+b) (since pulsar 2.1) the standalone should start bookkeeper table service as 
well.
+
+example output:
+
+```shell
+12:12:26.099 [main] INFO  org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble 
- 'default' namespace for table service : namespace_name: "default"
+default_stream_conf {
+  key_type: HASH
+  min_num_ranges: 24
+  initial_num_ranges: 24
+  split_policy {
+fixed_range_policy {
+  num_ranges: 2
+}
+  }
+  rolling_policy {
+size_policy {
+  max_segment_size: 134217728
+}
+  }
+  retention_policy {
+time_policy {
+  retention_minutes: -1
+}
+  }
+}
+```
+
+c) functions worker is started correctly.
+
+example output:
+
+```shell
+12:12:31.647 [pulsar-external-listener-70-1] INFO  
org.apache.pulsar.functions.worker.MembershipManager - Worker 
c-standalone-fw-localhost-6750:localhost:6750 became the leader.
+```
+
+b) sanity check before moving to next steps:
+
+```shell
+// check pulsar binary port is listened correctly
+$ telnet localhost 6650
+
+// check function cluster
+$ curl -s http://localhost:8080/admin/v2/functions/cluster
+// example output
+[{"workerId":"c-standalone-fw-localhost-6750","workerHostname":"localhost","port":6750}]
+
+// check brokers 
+$ curl -s http://localhost:8080/admin/v2/namespaces/public
+// example outoupt
+["public/default","public/functions"]
+
+// check connectors
+$ curl -s http://localhost:8080/admin/v2/functions/connectors
+// example output
+[{"name":"aerospike","description":"Aerospike database 
sink","sinkClass":"org.apache.pulsar.io.aerospike.AerospikeStringSink"},{"name":"cassandra","description":"Writes
 data into 
Cassandra","sinkClass":"org.apache.pulsar.io.cassandra.CassandraStringSink"},{"name":"kafka","description":"Kafka
 source and sink 
connector","sourceClass":"org.apache.pulsar.io.kafka.KafkaStringSource","sinkClass":"org.apache.pulsar.io.kafka.KafkaStringSink"},{"name":"kinesis","description":"Kinesis
 sink connect [...]
+
+// check table services
+$ telnet localhost 4181
+```
+
 2. Open another terminal to submit a Java Exclamation function.
 
 ```shell

[incubator-pulsar.wiki] branch master updated: Updated Home (markdown)

[jerrypeng]

This is an automated email from the ASF dual-hosted git repository.

jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new b9d839d  Updated Home (markdown)
b9d839d is described below

commit b9d839d6343ebbb9dc1ef856e82421be9b8c2a57
Author: Boyang Jerry Peng 
AuthorDate: Thu Jul 19 12:23:46 2018 -0700

Updated Home (markdown)
---
 Home.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Home.md b/Home.md
index e5eb497..d0d8ebf 100644
--- a/Home.md
+++ b/Home.md
@@ -5,6 +5,7 @@
 
 We encourage to document any big change or feature or any addition to public 
user APIs through a design document to be discussed with the community.
 
+* [PIP 18: Pulsar 
SQL](https://github.com/apache/incubator-pulsar/wiki/PIP-18:-Pulsar-SQL)
 * [PIP 17: Tiered storage for Pulsar 
topics](https://github.com/apache/incubator-pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics)
 * [PIP 16: Pulsar "instance" terminology 
change](https://github.com/apache/incubator-pulsar/wiki/PIP-16:-Pulsar-%22instance%22-terminology-change)
 * [PIP 15: Pulsar 
Functions](https://github.com/apache/incubator-pulsar/wiki/PIP-15:-Pulsar-Functions)

[incubator-pulsar.wiki] branch master updated: Created PIP 18 Pulsar SQL (markdown)

[jerrypeng]

This is an automated email from the ASF dual-hosted git repository.

jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new e5141ba  Created PIP 18 Pulsar SQL (markdown)
e5141ba is described below

commit e5141bacaa85297a074f662178b47add6648434c
Author: Boyang Jerry Peng 
AuthorDate: Thu Jul 19 12:22:51 2018 -0700

Created PIP 18 Pulsar SQL (markdown)
---
 PIP-18-Pulsar-SQL.md | 5 +
 1 file changed, 5 insertions(+)

diff --git a/PIP-18-Pulsar-SQL.md b/PIP-18-Pulsar-SQL.md
new file mode 100644
index 000..bee8d4f
--- /dev/null
+++ b/PIP-18-Pulsar-SQL.md
@@ -0,0 +1,5 @@
+* **Status**: In Progress
+* **Authors**: Jerry Peng
+* **Mailing List discussion**: 
+N/A
+* **Prototype**: https://github.com/jerrypeng/presto/tree/pulsar_connector

[incubator-pulsar.wiki] branch master updated: add instructions to unpack connectors

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 9f6557e  add instructions to unpack connectors
9f6557e is described below

commit 9f6557e16b49ded795512671c7bb53ea6811f743
Author: Sijie Guo 
AuthorDate: Thu Jul 19 12:11:57 2018 -0700

add instructions to unpack connectors
---
 Release-Candidate-Validation.md | 28 +++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
index 0813891..be734db 100644
--- a/Release-Candidate-Validation.md
+++ b/Release-Candidate-Validation.md
@@ -2,7 +2,33 @@ Here are some instructions for reviewing and validating a 
release candidate.
 
 ### Validate the binary distribution
 
-Unzip the binary distribution. The unzipped binary should be in a directory 
called `apache-pulsar-`. All the operations below happen within that 
directory.
+ Download the binary distributions
+
+Since Pulsar 2.1, we are shipping two binary distributions. one is server 
component, containing all the binaries for running Pulsar Service, the other 
one is a built-in connectors distribution, containing all the built-in 
connectors.
+
+Unzip the server distribution `apache-pulsar--bin.tar.gz`. The 
unzipped binary should be in a directory called `apache-pulsar-`. All 
the operations below happen within that directory.
+
+```shell
+$ cd apache-pulsar-
+```
+
+In the apache pulsar directory, unzip the io package and copy the connectors 
as `connectors` in the pulsar distribution.
+
+```shell
+$ tar xf /path/to/apache-pulsar-io-connectors--bin.tar.gz
+// you will find a directory named `apache-pulsar-io-connectors-`
+// move the connectors
+$ mv apache-pulsar-io-connectors-/connectors connectors
+$ ls connectors
+pulsar-io-aerospike-.nar
+pulsar-io-cassandra-.nar 
+pulsar-io-kafka-.nar 
+pulsar-io-kinesis-.nar   
+pulsar-io-rabbitmq-.nar  
+pulsar-io-twitter-.nar
+```
+
+ Validate Pub/Sub and Java Functions
 
 1. Open one terminal to start a standalone cluster.
 

[incubator-pulsar.wiki] branch master updated: Created Release Candidate Validation (markdown)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new b8edf85  Created Release Candidate Validation (markdown)
b8edf85 is described below

commit b8edf85c1151aaac92e8bb33ecfec02ecfedf8f7
Author: Sijie Guo 
AuthorDate: Thu Jul 19 12:00:55 2018 -0700

Created Release Candidate Validation (markdown)
---
 Release-Candidate-Validation.md | 119 
 1 file changed, 119 insertions(+)

diff --git a/Release-Candidate-Validation.md b/Release-Candidate-Validation.md
new file mode 100644
index 000..0813891
--- /dev/null
+++ b/Release-Candidate-Validation.md
@@ -0,0 +1,119 @@
+Here are some instructions for reviewing and validating a release candidate.
+
+### Validate the binary distribution
+
+Unzip the binary distribution. The unzipped binary should be in a directory 
called `apache-pulsar-`. All the operations below happen within that 
directory.
+
+1. Open one terminal to start a standalone cluster.
+
+```shell
+$ bin/pulsar standalone
+```
+
+2. Open another terminal to submit a Java Exclamation function.
+
+```shell
+$ bin/pulsar-admin functions create --functionConfigFile 
examples/example-function-config.yaml --jar examples/api-examples.jar
+```
+
+You should see `Created Successfully`.
+
+3. At the same terminal as step 2, retrieve the function config:
+
+```shell
+$ bin/pulsar-admin functions get --tenant test --namespace test-namespace 
--name example
+```
+
+example output:
+
+```shell
+{
+  "tenant": "test",
+  "namespace": "test-namespace",
+  "name": "example",
+  "className": "org.apache.pulsar.functions.api.examples.ExclamationFunction",
+  "userConfig": "{\"PublishTopic\":\"test_result\"}",
+  "autoAck": true,
+  "parallelism": 1,
+  "source": {
+"topicsToSerDeClassName": {
+  "test_src": ""
+},
+"typeClassName": "java.lang.String"
+  },
+  "sink": {
+"topic": "test_result",
+"typeClassName": "java.lang.String"
+  },
+  "resources": {}
+}
+```
+
+4. At the same terminal as step 3, retrieve the function status:
+
+```shell
+$ bin/pulsar-admin functions getstatus --tenant test --namespace 
test-namespace --name example
+```
+
+example output:
+
+```shell
+{
+  "functionStatusList": [
+{
+  "running": true,
+  "instanceId": "0",
+  "metrics": {
+"metrics": {
+  "__total_processed__": {},
+  "__total_successfully_processed__": {},
+  "__total_system_exceptions__": {},
+  "__total_user_exceptions__": {},
+  "__total_serialization_exceptions__": {},
+  "__avg_latency_ms__": {}
+}
+  },
+  "workerId": "c-standalone-fw-localhost-6750"
+}
+  ]
+}
+```
+
+5. At the same terminal as step 4, subscribe the output topic `test_result`.
+
+```shell
+$ bin/pulsar-client consume -s test-sub -n 0 test_result
+```
+
+6. Open a new terminal to produce messages into the input topic `test_src`.
+
+```shell
+$ bin/pulsar-client produce -m "test-messages-`date`" -n 10 test_src
+```
+
+7. At the terminal of step 5, you will see the messages produced by the 
Exclamation function.
+
+example output:
+
+```shell
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+- got message -
+test-messages-Thu Jul 19 11:59:15 PDT 2018!
+```
\ No newline at end of file

[GitHub] grantwwu commented on issue #2194: WIP: Destination fix

[GitBox]

grantwwu commented on issue #2194: WIP: Destination fix
URL: https://github.com/apache/incubator-pulsar/pull/2194#issuecomment-406359597
 
 
   Uh sorry I have more stuff to change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bardock commented on issue #2013: Unable to consume messages from a partition

[GitBox]

bardock commented on issue #2013: Unable to consume messages from a partition
URL: 
https://github.com/apache/incubator-pulsar/issues/2013#issuecomment-406349051
 
 
   @sijie We don't have bookkeeper's metrics. 
   
   This cluster has 9 bookies, and we see that 4 of them were using more CPU 
than the rest (3 of them were stuck at 13%). When we restarted the cluster, 
every bookie's CPU went normal.
   ![screen shot 2018-07-19 at 12 43 10 
pm](https://user-images.githubusercontent.com/1980715/42953792-910c4628-8b51-11e8-9bbe-0fc10969c757.png)
   The problem with partition-4 of this topic started at 16:05 (GMT -3).
   
   So we believe these bookies were performing poorly and caused the 
throttling. Sadly, we don't have a jstack for that moment. 
   Is there anything we can tune up or monitor to prevent this issue? Here is 
our [config 
file](https://github.com/apache/incubator-pulsar/files/2210966/bookkeeper.conf.txt).
   
   We are using bk 4.7.0 (shipped with pulsar 2.0.0-rc1) but we are planning to 
upgrade to 4.7.1.
   
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] 01/01: Release 2.1.0-incubating

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git

commit a3082c1954b189d75cc8b523156c7a480da6b618
Author: Sijie Guo 
AuthorDate: Thu Jul 19 10:05:06 2018 -0700

Release 2.1.0-incubating
---
 buildtools/pom.xml| 2 +-
 distribution/io/pom.xml   | 2 +-
 distribution/pom.xml  | 2 +-
 distribution/server/pom.xml   | 2 +-
 docker/grafana/pom.xml| 2 +-
 docker/pom.xml| 2 +-
 docker/pulsar-all/pom.xml | 2 +-
 docker/pulsar/pom.xml | 2 +-
 managed-ledger-shaded/pom.xml | 2 +-
 managed-ledger/pom.xml| 2 +-
 pom.xml   | 2 +-
 protobuf-shaded/pom.xml   | 2 +-
 pulsar-broker-auth-athenz/pom.xml | 2 +-
 pulsar-broker-common/pom.xml  | 2 +-
 pulsar-broker-shaded/pom.xml  | 2 +-
 pulsar-broker/pom.xml | 2 +-
 pulsar-client-admin-shaded/pom.xml| 2 +-
 pulsar-client-admin/pom.xml   | 2 +-
 pulsar-client-auth-athenz/pom.xml | 2 +-
 pulsar-client-kafka-compat/pom.xml| 2 +-
 pulsar-client-kafka-compat/pulsar-client-kafka-shaded/pom.xml | 2 +-
 pulsar-client-kafka-compat/pulsar-client-kafka-tests/pom.xml  | 2 +-
 pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml| 2 +-
 pulsar-client-schema/pom.xml  | 2 +-
 pulsar-client-shaded/pom.xml  | 2 +-
 pulsar-client-tools-test/pom.xml  | 2 +-
 pulsar-client-tools/pom.xml   | 2 +-
 pulsar-client/pom.xml | 2 +-
 pulsar-common/pom.xml | 2 +-
 pulsar-discovery-service/pom.xml  | 2 +-
 pulsar-functions/api-java/pom.xml | 2 +-
 pulsar-functions/instance/pom.xml | 2 +-
 pulsar-functions/java-examples/pom.xml| 2 +-
 pulsar-functions/metrics/pom.xml  | 2 +-
 pulsar-functions/pom.xml  | 2 +-
 pulsar-functions/proto-shaded/pom.xml | 2 +-
 pulsar-functions/proto/pom.xml| 2 +-
 pulsar-functions/runtime-all/pom.xml  | 2 +-
 pulsar-functions/runtime-shaded/pom.xml   | 2 +-
 pulsar-functions/runtime/pom.xml  | 2 +-
 pulsar-functions/utils/pom.xml| 2 +-
 pulsar-functions/worker/pom.xml   | 2 +-
 pulsar-io/aerospike/pom.xml   | 2 +-
 pulsar-io/cassandra/pom.xml   | 2 +-
 pulsar-io/core/pom.xml| 2 +-
 pulsar-io/kafka/pom.xml   | 2 +-
 pulsar-io/kinesis/pom.xml | 2 +-
 pulsar-io/pom.xml | 2 +-
 pulsar-io/rabbitmq/pom.xml| 2 +-
 pulsar-io/twitter/pom.xml | 2 +-
 pulsar-log4j2-appender/pom.xml| 2 +-
 pulsar-proxy/pom.xml  | 2 +-
 pulsar-spark/pom.xml  | 2 +-
 pulsar-storm/pom.xml  | 2 +-
 pulsar-testclient/pom.xml | 2 +-
 pulsar-websocket/pom.xml  | 2 +-
 pulsar-zookeeper-utils/pom.xml| 2 +-
 pulsar-zookeeper/pom.xml  | 2 +-
 tests/docker-images/latest-version-image/pom.xml  | 2 +-
 tests/docker-images/pom.xml   | 2 +-
 tests/integration-tests-base/pom.xml  | 2 +-
 tests/integration-tests-topologies/pom.xml| 2 +-
 tests/integration-tests-utils/pom.xml | 2 +-
 tests/integration/compaction/pom.xml  | 2 +-
 tests/integration/pom.xml | 2 +-
 tests/integration/s3-offload/pom.xml  | 2 +-
 tests/integration/semantics/pom.xml   

[incubator-pulsar] branch branch-2.1 updated (d3261ed -> a3082c1)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git.


omit d3261ed  Enable authentication and authorization for some admin APIs 
(#2046)
omit fde1da6  Fix license in binary distribution (#2040)
omit f66b74b  Fixed resetting of producer sequence id counter after 
receiving SendError before any message was successfully published (#2047)
omit 94484a5  Fixed lookups for v2 topics in C++ client lib with HTTP 
service URL (#2043)
omit 1d32b75  Add depedency on grpcio and protobuf for Python lib (#2044)
omit de7332b  Move AuthenticationFilter to pulsar-broker-common (#2039)
omit 30f6128  Exclude a few files from apache-rat:check (#2035)
omit 3fbe4ac  Added custom compiled OpenSSL into docker image for RPM/Deb
omit e959f09  Added integer types casting to avoid compile errors with 
gcc-4.8 (#2031)
omit 7f8985a  Fixed C99 variable initialization in cgo header (#2029)
omit 840ad88  Update LICENSE & NOTICE for newly introduced or 
version-upgraded dependencies  (#2023)
omit 10397f7  Remove jboss-reflect dependency (#2022)
omit 270dcf1  Release 2.1.0-incubating
 add 6eb238b  Remove jboss-reflect dependency (#2022)
 add 7a5072e  Update LICENSE & NOTICE for newly introduced or 
version-upgraded dependencies  (#2023)
 add 8a8bef7  Fixup misleading text in compaction cookbook (#2026)
 add 172f050  Added message parser to decode raw headers and payload 
without a Consumer instance (#2019)
 add f7c5696   adding pulsar io docs (#2001)
 add b506793  Helm charts for deployment on GKE (#1993)
 add 8b2929b  Bump Pulsar version to 2.2.0-incubating (#2027)
 add 97d06c9  Fixed C99 variable initialization in cgo header (#2029)
 add ae90f22  Change MessageProcessor Interface to be public (#2030)
 add bdf52bc  Added integer types casting to avoid compile errors with 
gcc-4.8 (#2031)
 add ca93d4a  Added custom compiled OpenSSL into docker image for RPM/Deb 
(#2032)
 add b0292a1   adding a module that shades managed-ledge (#2034)
 add 174fd02  Exclude a few files from apache-rat:check (#2035)
 add c80dfb6  Move AuthenticationFilter to pulsar-broker-common (#2039)
 add 7c30702  Fix license in binary distribution (#2040)
 add 4487d04  Add depedency on grpcio and protobuf for Python lib (#2044)
 add ae596ed  Inject AuthenticationService into ProxyService (#2038)
 add c16dfbe  Original principal authorization for admin API (#2028)
 add 398a0bd  Enable authentication and authorization for some admin APIs 
(#2046)
 add 583ddee  Fixed lookups for v2 topics in C++ client lib with HTTP 
service URL (#2043)
 add f1df048  Fixed resetting of producer sequence id counter after 
receiving SendError before any message was successfully published (#2047)
 add 99c0ed2  Issue 1416: Docs about garbage collection are 
inaccurate/lacking (#2049)
 add 4d1cf20  Introduce testcontainer based cluster integration tests 
(#2055)
 add bee124e  Add semantic integration tests for non-persistent topics and 
effectively-once producing (#2059)
 add f14b9e9  fix bug with specifying jar via config file (#2053)
 add 7cebe23  Forward user-properties to sink (#2057)
 add 89c7b65  Pulsar Kinesis-sink upgrade aws-sdk version (#2058)
 add 9b1cc78  Forward encryption properties with encrypted payload to 
consumer (#2024)
 add 60a07e9  Migrate CLI integration test mode to use testcontainers 
(#2060)
 add 124dfaf   Move integration tests for function upload/download to 
apache (#2061)
 add 5766844  fix bug in concerning ContextImpl (#2052)
 add 4ade1bd  Tests Python Wheel packaging in CI build (#2042)
 add b22e883  Package Pulsar IO connectors into separate tgz in form of nar 
archives (#2054)
 add d01cc75  Fix build issue in master (#2062)
 add 791b596  Add integration tests on pulsar functions on process mode 
(#2063)
 add 7a10d93  GCS offload support(1): rename s3offload related classes to 
be reuse-able (#2064)
 add 7559632  Improve RedeliverUnacknowledgedMessages binary protocol docs 
(#2066)
 add e16f91c  Run functions runtime test with thread mode (#2070)
 add 5e4579d  Pass encryption-context to pulsar-sink if source receive 
encrypted message (#2068)
 add 5614cc5  Integration test coverage on python functions (#2072)
 add 5d73bb4  Fix: destination lookup url for v2 (#2075)
 add 8e95ddc  By default PulsarSource consume encrypted message (#2074)
 add e422a6f  fix: configure crypto-action for internal consumer 
independently cryptoReader (#2073)
 add 9a8428a  shade dependencies in pulsar client and admin (#2069)
 add 59fc8e2  Dump container & pulsar log before container stops (#2076)
 add c9d1066  Fix python/CMakeLists.txt (#2084)
 add ec04dc0  Move Pulsar IO as a top-category section at the sidebar 
(#2050)
   

[GitHub] sijie closed pull request #2201: function-instance lookup: retrieve function instance owner's workerId

[GitBox]

sijie closed pull request #2201: function-instance lookup: retrieve function 
instance owner's workerId
URL: https://github.com/apache/incubator-pulsar/pull/2201
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java 
b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java
index 78fd2e2e40..a7b34ae857 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java
@@ -104,6 +104,7 @@
 final String tenant = "external-repl-prop";
 String pulsarFunctionsNamespace = tenant + "/use/pulsar-function-admin";
 String primaryHost;
+String workerId;
 
 private final int ZOOKEEPER_PORT = PortManager.nextFreePort();
 private final int brokerWebServicePort = PortManager.nextFreePort();
@@ -220,9 +221,9 @@ private WorkerService 
createPulsarFunctionWorker(ServiceConfiguration config) {
 workerConfig.setWorkerPort(workerServicePort);
 workerConfig.setPulsarFunctionsCluster(config.getClusterName());
 String hostname = 
ServiceConfigurationUtils.getDefaultOrConfiguredAddress(config.getAdvertisedAddress());
+this.workerId = "c-" + config.getClusterName() + "-fw-" + hostname + 
"-" + workerConfig.getWorkerPort();
 workerConfig.setWorkerHostname(hostname);
-workerConfig
-.setWorkerId("c-" + config.getClusterName() + "-fw-" + 
hostname + "-" + workerConfig.getWorkerPort());
+workerConfig.setWorkerId(workerId);
 
 
workerConfig.setClientAuthenticationPlugin(AuthenticationTls.class.getName());
 workerConfig.setClientAuthenticationParameters(
@@ -368,8 +369,10 @@ public void testPulsarSinkStats() throws Exception {
 
 double count = 
metricsData.get(JavaInstanceRunnable.METRICS_TOTAL_PROCESSED).getCount();
 double success = 
metricsData.get(JavaInstanceRunnable.METRICS_TOTAL_SUCCESS).getCount();
+String ownerWorkerId = stats.getWorkerId();
 Assert.assertEquals((int) count, totalMsgs);
 Assert.assertEquals((int) success, totalMsgs);
+Assert.assertEquals(ownerWorkerId, workerId);
 }
 
 protected FunctionDetails createSinkConfig(String jarFile, String tenant, 
String namespace, String functionName, String sinkTopic, String 
subscriptionName) {
diff --git a/pulsar-functions/proto/src/main/proto/InstanceCommunication.proto 
b/pulsar-functions/proto/src/main/proto/InstanceCommunication.proto
index 1a80f5ecf6..0078bc2544 100644
--- a/pulsar-functions/proto/src/main/proto/InstanceCommunication.proto
+++ b/pulsar-functions/proto/src/main/proto/InstanceCommunication.proto
@@ -50,6 +50,8 @@ message FunctionStatus {
 int64 lastInvocationTime = 13;
 string instanceId = 14;
 MetricsData metrics = 15;
+// owner of function-instance
+string workerId = 16;
 }
 
 message FunctionStatusList {
diff --git 
a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
 
b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
index bc6dbe3686..05a79d8da5 100644
--- 
a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
+++ 
b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
@@ -239,9 +239,10 @@ public synchronized void 
removeAssignments(Collection assignments) {
  */
 public InstanceCommunication.FunctionStatus 
getFunctionInstanceStatus(String tenant, String namespace,
   
String functionName, int instanceId) {
-String workerId = this.workerConfig.getWorkerId();
-
 Assignment assignment = this.findAssignment(tenant, namespace, 
functionName, instanceId);
+final String assignedWorkerId = assignment.getWorkerId();
+final String workerId = this.workerConfig.getWorkerId();
+
 if (assignment == null) {
 InstanceCommunication.FunctionStatus.Builder functionStatusBuilder
 = InstanceCommunication.FunctionStatus.newBuilder();
@@ -252,13 +253,16 @@ public synchronized void 
removeAssignments(Collection assignments) {
 
 InstanceCommunication.FunctionStatus functionStatus = null;
 // If I am running worker
-if (assignment.getWorkerId().equals(workerId)) {
+if (assignedWorkerId.equals(workerId)) {
 FunctionRuntimeInfo functionRuntimeInfo = 
this.getFunctionRuntimeInfo(
 
Utils.getFullyQualifiedInstanceId(assignment.getInstance()));
 

[incubator-pulsar] branch master updated: function-instance lookup: retrieve function instance owner's workerId (#2201)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 98eeaf5  function-instance lookup: retrieve function instance owner's 
workerId (#2201)
98eeaf5 is described below

commit 98eeaf5fb8344e021e1ff3a20c6c36759a6644bd
Author: Rajan Dhabalia 
AuthorDate: Thu Jul 19 09:50:46 2018 -0700

function-instance lookup: retrieve function instance owner's workerId 
(#2201)

### Motivation

for administrative purpose, we need to know worker-owner of the 
function-instance. so, adding workerId along with function-stats to find out 
owner of the instance.
---
 .../test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java  |  7 +--
 .../proto/src/main/proto/InstanceCommunication.proto   |  2 ++
 .../pulsar/functions/worker/FunctionRuntimeManager.java| 14 ++
 3 files changed, 17 insertions(+), 6 deletions(-)

diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java 
b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java
index 78fd2e2..a7b34ae 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java
@@ -104,6 +104,7 @@ public class PulsarSinkE2ETest {
 final String tenant = "external-repl-prop";
 String pulsarFunctionsNamespace = tenant + "/use/pulsar-function-admin";
 String primaryHost;
+String workerId;
 
 private final int ZOOKEEPER_PORT = PortManager.nextFreePort();
 private final int brokerWebServicePort = PortManager.nextFreePort();
@@ -220,9 +221,9 @@ public class PulsarSinkE2ETest {
 workerConfig.setWorkerPort(workerServicePort);
 workerConfig.setPulsarFunctionsCluster(config.getClusterName());
 String hostname = 
ServiceConfigurationUtils.getDefaultOrConfiguredAddress(config.getAdvertisedAddress());
+this.workerId = "c-" + config.getClusterName() + "-fw-" + hostname + 
"-" + workerConfig.getWorkerPort();
 workerConfig.setWorkerHostname(hostname);
-workerConfig
-.setWorkerId("c-" + config.getClusterName() + "-fw-" + 
hostname + "-" + workerConfig.getWorkerPort());
+workerConfig.setWorkerId(workerId);
 
 
workerConfig.setClientAuthenticationPlugin(AuthenticationTls.class.getName());
 workerConfig.setClientAuthenticationParameters(
@@ -368,8 +369,10 @@ public class PulsarSinkE2ETest {
 
 double count = 
metricsData.get(JavaInstanceRunnable.METRICS_TOTAL_PROCESSED).getCount();
 double success = 
metricsData.get(JavaInstanceRunnable.METRICS_TOTAL_SUCCESS).getCount();
+String ownerWorkerId = stats.getWorkerId();
 Assert.assertEquals((int) count, totalMsgs);
 Assert.assertEquals((int) success, totalMsgs);
+Assert.assertEquals(ownerWorkerId, workerId);
 }
 
 protected FunctionDetails createSinkConfig(String jarFile, String tenant, 
String namespace, String functionName, String sinkTopic, String 
subscriptionName) {
diff --git a/pulsar-functions/proto/src/main/proto/InstanceCommunication.proto 
b/pulsar-functions/proto/src/main/proto/InstanceCommunication.proto
index 1a80f5e..0078bc2 100644
--- a/pulsar-functions/proto/src/main/proto/InstanceCommunication.proto
+++ b/pulsar-functions/proto/src/main/proto/InstanceCommunication.proto
@@ -50,6 +50,8 @@ message FunctionStatus {
 int64 lastInvocationTime = 13;
 string instanceId = 14;
 MetricsData metrics = 15;
+// owner of function-instance
+string workerId = 16;
 }
 
 message FunctionStatusList {
diff --git 
a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
 
b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
index bc6dbe3..05a79d8 100644
--- 
a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
+++ 
b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
@@ -239,9 +239,10 @@ public class FunctionRuntimeManager implements 
AutoCloseable{
  */
 public InstanceCommunication.FunctionStatus 
getFunctionInstanceStatus(String tenant, String namespace,
   
String functionName, int instanceId) {
-String workerId = this.workerConfig.getWorkerId();
-
 Assignment assignment = this.findAssignment(tenant, namespace, 
functionName, instanceId);
+final String assignedWorkerId = assignment.getWorkerId();
+final String workerId = this.workerConfig.getWorkerId();
+
 if (assignment == null) {
 InstanceCommunication.FunctionStatus.Builder functionStatusBuilder
 

[GitHub] merlimat closed pull request #2202: attach shaded artifact while deploying to maven-repo

[GitBox]

merlimat closed pull request #2202: attach shaded artifact while deploying to 
maven-repo
URL: https://github.com/apache/incubator-pulsar/pull/2202
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pulsar-functions/runtime-all/pom.xml 
b/pulsar-functions/runtime-all/pom.xml
index b1ae4c724f..b7bdf412f6 100644
--- a/pulsar-functions/runtime-all/pom.xml
+++ b/pulsar-functions/runtime-all/pom.xml
@@ -75,6 +75,7 @@
 
   java-instance
   false
+  true
   
 
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: attach shaded artifact while deploying to maven-repo (#2202)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new dc06215  attach shaded artifact while deploying to maven-repo (#2202)
dc06215 is described below

commit dc0621527c14e630f2c9bc20e3bac1560bb0690d
Author: Rajan Dhabalia 
AuthorDate: Thu Jul 19 08:51:26 2018 -0700

attach shaded artifact while deploying to maven-repo (#2202)
---
 pulsar-functions/runtime-all/pom.xml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/pulsar-functions/runtime-all/pom.xml 
b/pulsar-functions/runtime-all/pom.xml
index b1ae4c7..b7bdf41 100644
--- a/pulsar-functions/runtime-all/pom.xml
+++ b/pulsar-functions/runtime-all/pom.xml
@@ -75,6 +75,7 @@
 
   java-instance
   false
+  true
   
 
 

[GitHub] merlimat commented on issue #2203: Make sure schema is initialized before the topic is loaded

[GitBox]

merlimat commented on issue #2203: Make sure schema is initialized before the 
topic is loaded
URL: https://github.com/apache/incubator-pulsar/pull/2203#issuecomment-406319247
 
 
   retest this please 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #2191: Removed shading relocations for Circe-checksum and lz4 libraries

[GitBox]

merlimat commented on issue #2191: Removed shading relocations for 
Circe-checksum and lz4 libraries
URL: https://github.com/apache/incubator-pulsar/pull/2191#issuecomment-406319104
 
 
   retest this please 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #2192: Rename Netty native libs to reflect shading prefix

[GitBox]

merlimat commented on issue #2192: Rename Netty native libs to reflect shading 
prefix
URL: https://github.com/apache/incubator-pulsar/pull/2192#issuecomment-406318823
 
 
   retest this please 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhaijack opened a new pull request #2205: Issue #2204: support v2 namespace in cpp client

[GitBox]

zhaijack opened a new pull request #2205: Issue #2204: support v2 namespace in 
cpp client
URL: https://github.com/apache/incubator-pulsar/pull/2205
 
 
   ### Motivation
   
   In pulsar-client-cpp, TopicName support V2 type, and contains method 
isV2Topic(), but NamespaceName not support it.
   We need to add support for NamespaceName to make it align with java client.
   
   ### Modifications
   
   - add v2 support in NamespaceName
   - add test case for it
   
   ### Result
   
   old behaviour not changed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dsambandam commented on issue #2195: Can't start Global Zookeeper in a Multi-cluster setup

[GitBox]

dsambandam commented on issue #2195: Can't start Global Zookeeper in a 
Multi-cluster setup 
URL: 
https://github.com/apache/incubator-pulsar/issues/2195#issuecomment-406296944
 
 
   Thanks @sijie  maybe documentation needs to be updated as well as the help. 
Quick Question - my configuration store in the meta data currently points to 
local zookeeper and I need to modify to point them to global zookeepers now. Is 
there a command to do this or do I have to delete the metadata and recreate the 
same?
   Any pointers for the right commands would be very helpful.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhaijack opened a new issue #2204: in cpp client, NamespaceName not support V2 type

[GitBox]

zhaijack opened a new issue #2204: in cpp client, NamespaceName not support V2 
type
URL: https://github.com/apache/incubator-pulsar/issues/2204
 
 
   In pulsar-client-cpp,  [TopicName](
   
https://github.com/apache/incubator-pulsar/blob/master/pulsar-client-cpp/lib/TopicName.h)
 support V2 type, and contains method `isV2Topic()`, but NamespaceName not 
support it. 
   
   We need to add support for NamespaceName.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on a change in pull request #1996: Cpp client: add multiTopicsConsumer

[GitBox]

ivankelly commented on a change in pull request #1996: Cpp client: add 
multiTopicsConsumer
URL: https://github.com/apache/incubator-pulsar/pull/1996#discussion_r203716660
 
 

 ##
 File path: pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc
 ##
 @@ -0,0 +1,621 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "MultiTopicsConsumerImpl.h"
+
+DECLARE_LOG_OBJECT()
+
+using namespace pulsar;
+
+MultiTopicsConsumerImpl::MultiTopicsConsumerImpl(ClientImplPtr client, const 
std::vector& topics,
+ const std::string& 
subscriptionName, TopicNamePtr topicName,
+ const ConsumerConfiguration& 
conf,
+ const LookupServicePtr 
lookupServicePtr)
+: client_(client),
+  subscriptionName_(subscriptionName),
+  topic_(topicName ? topicName->toString() : "EmptyTopics"),
+  conf_(conf),
+  state_(Pending),
+  messages_(1000),
+  listenerExecutor_(client->getListenerExecutorProvider()->get()),
+  messageListener_(conf.getMessageListener()),
+  namespaceName_(topicName ? topicName->getNamespaceName() : 
boost::shared_ptr()),
+  lookupServicePtr_(lookupServicePtr),
+  allTopicPartitionsNumber_(boost::make_shared>(0)),
+  topics_(topics) {
+std::stringstream consumerStrStream;
+consumerStrStream << "[Muti Topics Consumer: "
+  << "TopicName - " << topic_ << " - Subscription - " << 
subscriptionName << "]";
+consumerStr_ = consumerStrStream.str();
+
+if (conf.getUnAckedMessagesTimeoutMs() != 0) {
+unAckedMessageTrackerPtr_.reset(
+new 
UnAckedMessageTrackerEnabled(conf.getUnAckedMessagesTimeoutMs(), client, 
*this));
+} else {
+unAckedMessageTrackerPtr_.reset(new UnAckedMessageTrackerDisabled());
+}
+}
+
+void MultiTopicsConsumerImpl::start() {
+if (topics_.empty()) {
+setState(Ready);
+LOG_DEBUG("No topics passed in when create MultiTopicsConsumer.");
+multiTopicsConsumerCreatedPromise_.setValue(shared_from_this());
+return;
+}
+
+// start call subscribeOneTopicAsync for each single topic
+int topicsNumber = topics_.size();
+boost::shared_ptr> topicsNeedCreate = 
boost::make_shared>(topicsNumber);
+// subscribe for each passed in topic
+for (std::vector::const_iterator itr = topics_.begin(); itr 
!= topics_.end(); itr++) {
+subscribeOneTopicAsync(*itr).addListener(
+boost::bind(::handleOneTopicSubscribed, 
shared_from_this(), _1, _2, *itr,
+topicsNeedCreate));
+}
+}
+
+void MultiTopicsConsumerImpl::handleOneTopicSubscribed(Result result, Consumer 
consumer,
+   const std::string& 
topic,
+   
boost::shared_ptr> topicsNeedCreate) {
+int previous = topicsNeedCreate->fetch_sub(1);
+assert(previous > 0);
+
+if (result != ResultOk) {
+state_ = Failed;
+multiTopicsConsumerCreatedPromise_.setFailed(result);
+// unsubscribed all of the successfully subscribed partitioned 
consumers
+ResultCallback nullCallbackForCleanup = NULL;
+closeAsync(nullCallbackForCleanup);
+LOG_ERROR("Unable to create Consumer - "
+  << " Error - " << result);
+return;
+}
+
+LOG_DEBUG("Successfully Subscribed to topic " << topic << " in 
TopicsConsumer ");
+
+if (topicsNeedCreate->load() == 0) {
+LOG_INFO("Successfully Subscribed to Topics");
+setState(Ready);
+if (!namespaceName_) {
+namespaceName_ = TopicName::get(topic)->getNamespaceName();
+}
+multiTopicsConsumerCreatedPromise_.setValue(shared_from_this());
+return;
+}
+}
+
+// subscribe for passed in topic
+Future MultiTopicsConsumerImpl::subscribeOneTopicAsync(const 
std::string& topic) {
+TopicNamePtr topicName;
+ConsumerSubResultPromisePtr topicPromise = 
boost::make_shared>();
+if (!(topicName = TopicName::get(topic))) {
+LOG_ERROR("TopicName invalid: " << topic);
+  

[GitHub] zhaijack edited a comment on issue #1996: Cpp client: add multiTopicsConsumer

[GitBox]

zhaijack edited a comment on issue #1996: Cpp client: add multiTopicsConsumer
URL: https://github.com/apache/incubator-pulsar/pull/1996#issuecomment-406253189
 
 
   retest this please
   
   for `Jenkins: C++ / Python Tests` timeout


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhaijack removed a comment on issue #1996: Cpp client: add multiTopicsConsumer

[GitBox]

zhaijack removed a comment on issue #1996: Cpp client: add multiTopicsConsumer
URL: https://github.com/apache/incubator-pulsar/pull/1996#issuecomment-406159737
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhaijack commented on issue #1996: Cpp client: add multiTopicsConsumer

[GitBox]

zhaijack commented on issue #1996: Cpp client: add multiTopicsConsumer
URL: https://github.com/apache/incubator-pulsar/pull/1996#issuecomment-406253189
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on issue #2180: Cleanup Arquillian consolidate projects

[GitBox]

aahmed-se commented on issue #2180: Cleanup Arquillian consolidate projects
URL: https://github.com/apache/incubator-pulsar/pull/2180#issuecomment-406205039
 
 
   ```
   [ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 
63.982 s <<< FAILURE! - in 
org.apache.pulsar.tests.integration.offload.TestS3Offload
   [ERROR] 
testPublishOffloadAndConsumeViaCLI(org.apache.pulsar.tests.integration.offload.TestS3Offload)
  Time elapsed: 10.634 s  <<< FAILURE!
   org.apache.pulsar.client.admin.PulsarAdminException: 
   A MultiException has 5 exceptions.  They are:
   1. java.lang.NoSuchMethodError: 
org.apache.pulsar.common.util.ObjectMapperFactory.create()Lcom/fasterxml/jackson/databind/ObjectMapper;
   2. java.lang.IllegalStateException: Unable to perform operation: create on 
org.apache.pulsar.client.admin.internal.JacksonConfigurator
   3. java.lang.IllegalStateException: Unable to perform operation: create on 
org.glassfish.jersey.internal.ContextResolverFactory
   4. java.lang.IllegalStateException: Unable to perform operation: create on 
org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide
   5. java.lang.IllegalStateException: Unable to perform operation: create on 
org.glassfish.jersey.message.internal.MessageBodyFactory
   
at 
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:187)
at 
org.apache.pulsar.client.admin.internal.TopicsImpl$10.failed(TopicsImpl.java:436)
at 
org.glassfish.jersey.client.JerseyInvocation$7.failed(JerseyInvocation.java:965)
at 
org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:200)
at 
org.glassfish.jersey.client.ClientRuntime.access$500(ClientRuntime.java:74)
at 
org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:182)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
at 
org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:208)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
   Caused by: A MultiException has 5 exceptions.  They are:
   1. java.lang.NoSuchMethodError: 
org.apache.pulsar.common.util.ObjectMapperFactory.create()Lcom/fasterxml/jackson/databind/ObjectMapper;
   2. java.lang.IllegalStateException: Unable to perform operation: create on 
org.apache.pulsar.client.admin.internal.JacksonConfigurator
   3. java.lang.IllegalStateException: Unable to perform operation: create on 
org.glassfish.jersey.internal.ContextResolverFactory
   4. java.lang.IllegalStateException: Unable to perform operation: create on 
org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide
   5. java.lang.IllegalStateException: Unable to perform operation: create on 
org.glassfish.jersey.message.internal.MessageBodyFactory
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2180: Cleanup Arquillian consolidate projects

[GitBox]

sijie commented on issue #2180: Cleanup Arquillian consolidate projects
URL: https://github.com/apache/incubator-pulsar/pull/2180#issuecomment-406185295
 
 
   @aahmed-se can you attach those exceptions in this PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed pull request #2197: unshade avro in pulsar-client-schema

[GitBox]

sijie closed pull request #2197: unshade avro in pulsar-client-schema
URL: https://github.com/apache/incubator-pulsar/pull/2197
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pulsar-client-schema/pom.xml b/pulsar-client-schema/pom.xml
index a7d9f470d5..7ee5646aeb 100644
--- a/pulsar-client-schema/pom.xml
+++ b/pulsar-client-schema/pom.xml
@@ -285,8 +285,20 @@
 
 org.apache.avro
 
org.apache.pulsar.shade.org.apache.avro
-
-
+
+
org.apache.avro.reflect.AvroAlias
+
org.apache.avro.reflect.AvroDefault
+
org.apache.avro.reflect.AvroEncode
+
org.apache.avro.reflect.AvroIgnore
+
org.apache.avro.reflect.AvroMeta
+
org.apache.avro.reflect.AvroName
+
org.apache.avro.reflect.AvroSchema
+
org.apache.avro.reflect.Nullable
+
org.apache.avro.reflect.Stringable
+
org.apache.avro.reflect.Union
+
+
+ 
 
 org.codehaus.jackson
 
org.apache.pulsar.shade.org.codehaus.jackson


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: unshade avro in pulsar-client-schema (#2197)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 266834e  unshade avro in pulsar-client-schema (#2197)
266834e is described below

commit 266834ef2bde5ed2965964f818a369200857acf1
Author: Boyang Jerry Peng 
AuthorDate: Thu Jul 19 00:30:08 2018 -0700

unshade avro in pulsar-client-schema (#2197)

### Motivation

If users want to use Avro annotations with their POJOs, currently they 
would have to use the our shaded version of avro because we shade avro which is 
not the best user experience.  Ideally they should just use the original avro 
annotations

For example users want to something like this:
@org.apache.avro.reflect.AvroSchema(“{ \“type\“: \“long\“, \“logicalType\“: 
\“timestamp-millis\” }“)
   private long date;

but since we shade avro the user would need to specify like this:
@org.apache.pulsar.shade.org.apache.avro.reflect.AvroSchema(“{ \“type\“: 
\“long\“, \“logicalType\“: \“timestamp-millis\” }“)
   private long date;

### Modifications
Unshade avro in pulsar-client-schema
---
 pulsar-client-schema/pom.xml | 16 ++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/pulsar-client-schema/pom.xml b/pulsar-client-schema/pom.xml
index a7d9f47..7ee5646 100644
--- a/pulsar-client-schema/pom.xml
+++ b/pulsar-client-schema/pom.xml
@@ -285,8 +285,20 @@
 
 org.apache.avro
 
org.apache.pulsar.shade.org.apache.avro
-
-
+
+
org.apache.avro.reflect.AvroAlias
+
org.apache.avro.reflect.AvroDefault
+
org.apache.avro.reflect.AvroEncode
+
org.apache.avro.reflect.AvroIgnore
+
org.apache.avro.reflect.AvroMeta
+
org.apache.avro.reflect.AvroName
+
org.apache.avro.reflect.AvroSchema
+
org.apache.avro.reflect.Nullable
+
org.apache.avro.reflect.Stringable
+
org.apache.avro.reflect.Union
+
+
+ 
 
 org.codehaus.jackson
 
org.apache.pulsar.shade.org.codehaus.jackson

[GitHub] rdhabalia removed a comment on issue #2201: function-instance lookup: retrieve function instance owner's workerId

[GitBox]

rdhabalia removed a comment on issue #2201: function-instance lookup: retrieve 
function instance owner's workerId
URL: https://github.com/apache/incubator-pulsar/pull/2201#issuecomment-406180525
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia commented on issue #2201: function-instance lookup: retrieve function instance owner's workerId

[GitBox]

rdhabalia commented on issue #2201: function-instance lookup: retrieve function 
instance owner's workerId
URL: https://github.com/apache/incubator-pulsar/pull/2201#issuecomment-406180525
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on issue #2180: Cleanup Arquillian consolidate projects

[GitBox]

aahmed-se commented on issue #2180: Cleanup Arquillian consolidate projects
URL: https://github.com/apache/incubator-pulsar/pull/2180#issuecomment-406175783
 
 
   Pulsar client is throwing method not found exception in few of the tests.
   
   -Ali
   
   On Wed, Jul 18, 2018 at 11:45 PM Sijie Guo  wrote:
   
   > @aahmed-se  what are shading issues? what
   > errors have you been observed?
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   
   
   -- 
   -Ali
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat opened a new pull request #2203: Make sure schema is initialized before the topic is loaded

[GitBox]

merlimat opened a new pull request #2203: Make sure schema is initialized 
before the topic is loaded
URL: https://github.com/apache/incubator-pulsar/pull/2203
 
 
   ### Motivation
   
   There is a race condition when the topic creation is triggered by multiple 
producers at the same with a schema definition. 
   
1. The topic is created correctly and then each producers tries to add the 
schema
2. One single operation will first create the z-node entry for schema 
registry (that points to no schema) and then will add the schema
3. Other producers might happen to see the z-node but content will be 
invalid (ledgerId=-1)
   
   ### Modifications
   
* Do the `topic.addSchema()` operation *before* the topic is visible. That 
will ensure the whole chain of "schema" operations will be done at least once 
before the topic is finally loaded. After that, each producer/consumer will see 
at least a valid schema. 
   
   ### Result
   This fixes the flakyness in `PartitionedTopicsSchemaTest`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia removed a comment on issue #2201: function-instance lookup: retrieve function instance owner's workerId

[GitBox]

rdhabalia removed a comment on issue #2201: function-instance lookup: retrieve 
function instance owner's workerId
URL: https://github.com/apache/incubator-pulsar/pull/2201#issuecomment-406172636
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia commented on issue #2201: function-instance lookup: retrieve function instance owner's workerId

[GitBox]

rdhabalia commented on issue #2201: function-instance lookup: retrieve function 
instance owner's workerId
URL: https://github.com/apache/incubator-pulsar/pull/2201#issuecomment-406172636
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia commented on issue #2202: attach shaded artifact while deploying to maven-repo

[GitBox]

rdhabalia commented on issue #2202: attach shaded artifact while deploying to 
maven-repo
URL: https://github.com/apache/incubator-pulsar/pull/2202#issuecomment-406172513
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia removed a comment on issue #2202: attach shaded artifact while deploying to maven-repo

[GitBox]

rdhabalia removed a comment on issue #2202: attach shaded artifact while 
deploying to maven-repo
URL: https://github.com/apache/incubator-pulsar/pull/2202#issuecomment-406172513
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2180: Cleanup Arquillian consolidate projects

[GitBox]

sijie commented on issue #2180: Cleanup Arquillian consolidate projects
URL: https://github.com/apache/incubator-pulsar/pull/2180#issuecomment-406171993
 
 
   @aahmed-se what are shading issues? what errors have you been observed?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia commented on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

rdhabalia commented on issue #2200: support subscription-type to be passed in 
sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406171768
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia removed a comment on issue #2200: support subscription-type to be passed in sink-function

[GitBox]

rdhabalia removed a comment on issue #2200: support subscription-type to be 
passed in sink-function
URL: https://github.com/apache/incubator-pulsar/pull/2200#issuecomment-406171768
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Improved dashboard documentation with warning on advertised address (#2196)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new ffe3849  Improved dashboard documentation with warning on advertised 
address (#2196)
ffe3849 is described below

commit ffe38490fb22ed539c9b542fab97bea000790844
Author: Matteo Merli 
AuthorDate: Wed Jul 18 23:22:09 2018 -0700

Improved dashboard documentation with warning on advertised address (#2196)

### Motivation

The dashboard docs should warn user of setting the proper advertised 
address on standalone service to make it reachable from the stats collector 
inside Docker.
---
 site/docs/latest/admin/Dashboard.md| 14 +-
 site/docs/latest/getting-started/docker.md |  2 +-
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/site/docs/latest/admin/Dashboard.md 
b/site/docs/latest/admin/Dashboard.md
index 66cc68f..0516c58 100644
--- a/site/docs/latest/admin/Dashboard.md
+++ b/site/docs/latest/admin/Dashboard.md
@@ -36,7 +36,7 @@ The easiest way to use the dashboard is to run it inside a 
[Docker](https://www.
 To generate the Docker image:
 
 ```shell
-$ docker build -t pulsar-dashboard dashboard
+$ docker build -t apachepulsar/pulsar-dashboard dashboard
 ```
 
 To run the dashboard:
@@ -52,6 +52,18 @@ You need to specify only one service URL for a Pulsar 
cluster. Internally, the c
 
 Once the Docker container is running, the web dashboard will be accessible via 
`localhost` or whichever host is being used by Docker.
 
+{% include admonition.html type="warning" content="The `SERVICE_URL` that the 
dashboard uses needs to be reachable from inside the Docker container" %}
+
+If the Pulsar service is running in standalone mode in `localhost`, the 
`SERVICE_URL` would have to
+be the IP of the machine.
+
+Similarly, given the Pulsar standalone advertises itself with localhost by 
default, we need to
+explicitely set the advertise address to the host IP. For example:
+
+```shell
+$ bin/pulsar standalone --advertised-address 1.2.3.4
+```
+
 ### Known issues
 
 Pulsar [authentication](../../security/overview#authentication-providers) is 
not supported at this point. The dashboard's data collector does not pass any 
authentication-related data and will be denied access if the Pulsar broker 
requires authentication.
diff --git a/site/docs/latest/getting-started/docker.md 
b/site/docs/latest/getting-started/docker.md
index 526494e..6e1220f 100644
--- a/site/docs/latest/getting-started/docker.md
+++ b/site/docs/latest/getting-started/docker.md
@@ -123,7 +123,7 @@ You can find detailed documentation of all the APIs in the 
[Admin API Overview](
 In the simplest example, you can use curl to probe the stats for a particular 
topic:
 
 ```shell
-$ curl http://localhost:8080/admin/persistent/public/default/my-topic/stats | 
python -m json.tool
+$ curl http://localhost:8080/admin/v2/persistent/public/default/my-topic/stats 
| python -m json.tool
 ```
 
 The output will be something like this:

[GitHub] sijie closed pull request #2196: Improved dashboard documentation with warning on advertised address

[GitBox]

sijie closed pull request #2196: Improved dashboard documentation with warning 
on advertised address
URL: https://github.com/apache/incubator-pulsar/pull/2196
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/site/docs/latest/admin/Dashboard.md 
b/site/docs/latest/admin/Dashboard.md
index 66cc68f39e..0516c58fb0 100644
--- a/site/docs/latest/admin/Dashboard.md
+++ b/site/docs/latest/admin/Dashboard.md
@@ -36,7 +36,7 @@ The easiest way to use the dashboard is to run it inside a 
[Docker](https://www.
 To generate the Docker image:
 
 ```shell
-$ docker build -t pulsar-dashboard dashboard
+$ docker build -t apachepulsar/pulsar-dashboard dashboard
 ```
 
 To run the dashboard:
@@ -52,6 +52,18 @@ You need to specify only one service URL for a Pulsar 
cluster. Internally, the c
 
 Once the Docker container is running, the web dashboard will be accessible via 
`localhost` or whichever host is being used by Docker.
 
+{% include admonition.html type="warning" content="The `SERVICE_URL` that the 
dashboard uses needs to be reachable from inside the Docker container" %}
+
+If the Pulsar service is running in standalone mode in `localhost`, the 
`SERVICE_URL` would have to
+be the IP of the machine.
+
+Similarly, given the Pulsar standalone advertises itself with localhost by 
default, we need to
+explicitely set the advertise address to the host IP. For example:
+
+```shell
+$ bin/pulsar standalone --advertised-address 1.2.3.4
+```
+
 ### Known issues
 
 Pulsar [authentication](../../security/overview#authentication-providers) is 
not supported at this point. The dashboard's data collector does not pass any 
authentication-related data and will be denied access if the Pulsar broker 
requires authentication.
diff --git a/site/docs/latest/getting-started/docker.md 
b/site/docs/latest/getting-started/docker.md
index 526494e7cc..6e1220f70c 100644
--- a/site/docs/latest/getting-started/docker.md
+++ b/site/docs/latest/getting-started/docker.md
@@ -123,7 +123,7 @@ You can find detailed documentation of all the APIs in the 
[Admin API Overview](
 In the simplest example, you can use curl to probe the stats for a particular 
topic:
 
 ```shell
-$ curl http://localhost:8080/admin/persistent/public/default/my-topic/stats | 
python -m json.tool
+$ curl http://localhost:8080/admin/v2/persistent/public/default/my-topic/stats 
| python -m json.tool
 ```
 
 The output will be something like this:


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services