[GitHub] merlimat opened a new pull request #1804: Ensure BookKeeperClientFactory is only instantiated once in PulsarService

[GitBox]

merlimat opened a new pull request #1804: Ensure BookKeeperClientFactory is 
only instantiated once in PulsarService
URL: https://github.com/apache/incubator-pulsar/pull/1804
 
 
   ### Motivation
   
   Since SchemaRegistry also uses the BK client factory, 2 instances are being 
created and the ZK cache instance (owned by the BK client factory) is not being 
shutdown. 
   
   In unit tests, that means some threads are leaked and it's making difficult 
to repeat tests a lot of times when reproducing flaky tests.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1763: Document Python dependencies

[GitBox]

merlimat closed pull request #1763: Document Python dependencies
URL: https://github.com/apache/incubator-pulsar/pull/1763
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/site/_data/deps.yaml b/site/_data/deps.yaml
new file mode 100644
index 00..3717736c31
--- /dev/null
+++ b/site/_data/deps.yaml
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+- pulsar-client
+- protobuf
+- futures
+- grpcio
+- grpcio-tools
\ No newline at end of file
diff --git a/site/_includes/python-deps.html b/site/_includes/python-deps.html
new file mode 100644
index 00..e7f7d29bdf
--- /dev/null
+++ b/site/_includes/python-deps.html
@@ -0,0 +1,29 @@
+
+
+  {% for dep in site.data.deps %}
+  
+https://pypi.org/project/{{ dep }}">
+  {{ dep }}
+
+  
+  {% endfor %}
+
\ No newline at end of file
diff --git a/site/docs/latest/functions/api.md 
b/site/docs/latest/functions/api.md
index 4f92012886..c9105f533e 100644
--- a/site/docs/latest/functions/api.md
+++ b/site/docs/latest/functions/api.md
@@ -506,14 +506,15 @@ Writing Pulsar Functions in Python entails implementing 
one of two things:
 
 ### Getting started
 
-The requirements for writing Pulsar Functions in Python depend on your 
[deployment mode](../deployment):
+Regardless of which [deployment mode](../deployment) you're using, you'll need 
to install the following Python libraries on any machine that's running Pulsar 
Functions written in Python:
 
-* If you're writing a [Python native function](#python-native), you won't need 
to install any external dependencies
-* If you're writing a [Python SDK function](#python-sdk), you'll need to 
install the the [`pulsar-client`](/api/python) Python library.
+{% include python-deps.html %}
 
-  ```bash
-  $ pip install pulsar-client=={{ site.python_latest }}
-  ```
+That could be your local machine for [local run mode](../deployment#local-run) 
or a machine running a Pulsar {% popover broker %} for [cluster 
mode](../deployment#cluster-mode). To install those libraries using pip:
+
+```bash
+$ pip install {% for dep in site.data.deps %}{% if forloop.last %}{{ dep }}{% 
else %}{{ dep }} {% endif %}{% endfor %}
+```
 
 ### Packaging
 
diff --git a/site/docs/latest/functions/quickstart.md 
b/site/docs/latest/functions/quickstart.md
index 2b6eed4f44..ea51029782 100644
--- a/site/docs/latest/functions/quickstart.md
+++ b/site/docs/latest/functions/quickstart.md
@@ -212,6 +212,14 @@ If you see `Deleted successfully` in the output, then 
you've succesfully run, up
 
 ## Writing and running a new function
 
+{% include admonition.html type="info" content="
+In order to write and run the [Python](../api#python) function below, you'll 
need to install a few dependencies:
+
+```bash
+$ pip install pulsar-client protobuf futures grpcio grpcio-tools
+```
+" %}
+
 In the above examples, we ran and managed a pre-written Pulsar Function and 
saw how it worked. To really get our hands dirty, let's write and our own 
function from scratch, using the Python API. This simple function will also 
take a string as input but it will reverse the string and publish the 
resulting, reversed string to the specified topic.
 
 First, create a new Python file:


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Document Python dependencies (#1763)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new e6bcae5  Document Python dependencies (#1763)
e6bcae5 is described below

commit e6bcae5428a122a2fcd51f59731191eff1712ecb
Author: Luc Perkins 
AuthorDate: Thu May 17 16:30:12 2018 -0700

Document Python dependencies (#1763)

* add installation instructions for Python libs

* update list of python dependencies

* add pulsar-client lib to deps list

* add missing license headers

* add missing line to deps.yaml file under license header
---
 site/_data/deps.yaml | 24 
 site/_includes/python-deps.html  | 29 +
 site/docs/latest/functions/api.md| 13 +++--
 site/docs/latest/functions/quickstart.md |  8 
 4 files changed, 68 insertions(+), 6 deletions(-)

diff --git a/site/_data/deps.yaml b/site/_data/deps.yaml
new file mode 100644
index 000..3717736
--- /dev/null
+++ b/site/_data/deps.yaml
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+- pulsar-client
+- protobuf
+- futures
+- grpcio
+- grpcio-tools
\ No newline at end of file
diff --git a/site/_includes/python-deps.html b/site/_includes/python-deps.html
new file mode 100644
index 000..e7f7d29
--- /dev/null
+++ b/site/_includes/python-deps.html
@@ -0,0 +1,29 @@
+
+
+  {% for dep in site.data.deps %}
+  
+https://pypi.org/project/{{ dep }}">
+  {{ dep }}
+
+  
+  {% endfor %}
+
\ No newline at end of file
diff --git a/site/docs/latest/functions/api.md 
b/site/docs/latest/functions/api.md
index 4f92012..c9105f5 100644
--- a/site/docs/latest/functions/api.md
+++ b/site/docs/latest/functions/api.md
@@ -506,14 +506,15 @@ Writing Pulsar Functions in Python entails implementing 
one of two things:
 
 ### Getting started
 
-The requirements for writing Pulsar Functions in Python depend on your 
[deployment mode](../deployment):
+Regardless of which [deployment mode](../deployment) you're using, you'll need 
to install the following Python libraries on any machine that's running Pulsar 
Functions written in Python:
 
-* If you're writing a [Python native function](#python-native), you won't need 
to install any external dependencies
-* If you're writing a [Python SDK function](#python-sdk), you'll need to 
install the the [`pulsar-client`](/api/python) Python library.
+{% include python-deps.html %}
 
-  ```bash
-  $ pip install pulsar-client=={{ site.python_latest }}
-  ```
+That could be your local machine for [local run mode](../deployment#local-run) 
or a machine running a Pulsar {% popover broker %} for [cluster 
mode](../deployment#cluster-mode). To install those libraries using pip:
+
+```bash
+$ pip install {% for dep in site.data.deps %}{% if forloop.last %}{{ dep }}{% 
else %}{{ dep }} {% endif %}{% endfor %}
+```
 
 ### Packaging
 
diff --git a/site/docs/latest/functions/quickstart.md 
b/site/docs/latest/functions/quickstart.md
index 2b6eed4..ea51029 100644
--- a/site/docs/latest/functions/quickstart.md
+++ b/site/docs/latest/functions/quickstart.md
@@ -212,6 +212,14 @@ If you see `Deleted successfully` in the output, then 
you've succesfully run, up
 
 ## Writing and running a new function
 
+{% include admonition.html type="info" content="
+In order to write and run the [Python](../api#python) function below, you'll 
need to install a few dependencies:
+
+```bash
+$ pip install pulsar-client protobuf futures grpcio grpcio-tools
+```
+" %}
+
 In the above examples, we ran and managed a pre-written Pulsar Function and 
saw how it worked. To really get our hands dirty, let's write and our own 
function from scratch, using the Python API. This simple function will also 
take a string as input but it will reverse the string and publish the 
resulting, reversed string to the specified topic.
 
 First, create a new Python file:

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[incubator-pulsar] branch master updated: Added docker files and script to build static client library and Python bindings on Linux (#1793)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 1a0c8dc  Added docker files and script to build static client library 
and Python bindings on Linux (#1793)
1a0c8dc is described below

commit 1a0c8dcea29feb3d2cee71e4fcc3cab6b06e65bf
Author: Matteo Merli 
AuthorDate: Thu May 17 16:29:53 2018 -0700

Added docker files and script to build static client library and Python 
bindings on Linux (#1793)

* Added docker files and script to build static client library and Python 
bindings on Linux

* Added license headers
---
 pulsar-client-cpp/docker/Dockerfile| 144 +
 .../docker/build-client-lib-within-docker.sh   |  34 +
 pulsar-client-cpp/docker/build-client-lib.sh   |  44 +++
 .../docker/build-wheel-file-within-docker.sh   |  35 +
 pulsar-client-cpp/docker/build-wheels.sh   |  48 +++
 pulsar-client-cpp/docker/create-images.sh  |  48 +++
 pulsar-client-cpp/docker/push-images.sh|  51 
 7 files changed, 404 insertions(+)

diff --git a/pulsar-client-cpp/docker/Dockerfile 
b/pulsar-client-cpp/docker/Dockerfile
new file mode 100644
index 000..1df9545
--- /dev/null
+++ b/pulsar-client-cpp/docker/Dockerfile
@@ -0,0 +1,144 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+
+FROM quay.io/pypa/manylinux1_x86_64
+
+RUN yum install -y gtest-devel
+
+ARG PYTHON_VERSION
+ARG PYTHON_SPEC
+
+ENV PYTHON_VERSION=${PYTHON_VERSION}
+ENV PYTHON_SPEC=${PYTHON_SPEC}
+
+ENV PATH="/opt/python/${PYTHON_SPEC}/bin:${PATH}"
+
+RUN ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m 
/opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}
+
+
+# These dependencies can be found in Ubuntu but they're not compiled with 
-fPIC,
+# so they cannot be statically linked into a shared library
+
+
+# ZLib
+RUN curl -O -L https://zlib.net/zlib-1.2.11.tar.gz && \
+tar xvfz zlib-1.2.11.tar.gz && \
+cd zlib-1.2.11 && \
+CFLAGS="-fPIC -O3" ./configure && \
+make && make install && \
+rm -rf /zlib-1.2.11.tar.gz /zlib-1.2.11
+
+# Compile OpenSSL
+RUN curl -O -L 
https://github.com/openssl/openssl/archive/OpenSSL_1_0_2l.tar.gz && \
+tar xvfz OpenSSL_1_0_2l.tar.gz && \
+cd openssl-OpenSSL_1_0_2l/ && \
+./Configure -fPIC no-shared linux-x86_64 && \
+make && make install && \
+rm -rf /OpenSSL_1_0_2l.tar.gz /openssl-OpenSSL_1_0_2l
+
+# Download and compile boost
+RUN curl -O -L 
https://dl.bintray.com/boostorg/release/1.64.0/source/boost_1_64_0.tar.gz && \
+tar xvfz boost_1_64_0.tar.gz && \
+cd /boost_1_64_0 && \
+./bootstrap.sh 
--with-libraries=program_options,filesystem,regex,thread,system,python && \
+./b2 address-model=64 cxxflags=-fPIC link=static threading=multi 
variant=release install && \
+rm -rf /boost_1_64_0.tar.gz /boost_1_64_0
+
+# Download and copile protoubf
+RUN curl -O -L  
https://github.com/google/protobuf/releases/download/v3.3.0/protobuf-cpp-3.3.0.tar.gz
 && \
+tar xvfz protobuf-cpp-3.3.0.tar.gz && \
+cd protobuf-3.3.0/ && \
+CXXFLAGS=-fPIC ./configure && \
+make && make install && ldconfig && \
+rm -rf /protobuf-cpp-3.3.0.tar.gz /protobuf-3.3.0
+
+# Compile APR
+RUN curl -O -L  http://archive.apache.org/dist/apr/apr-1.5.2.tar.gz && \
+tar xvfz apr-1.5.2.tar.gz && \
+cd apr-1.5.2 && \
+CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure && \
+make && make install && \
+rm -rf /apr-1.5.2.tar.gz /apr-1.5.2
+
+# Compile APR-Util
+RUN curl -O -L  http://archive.apache.org/dist/apr/apr-util-1.5.4.tar.gz && \
+tar xvfz apr-util-1.5.4.tar.gz && \
+cd apr-util-1.5.4 && \
+CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure -with-apr=/usr/local/apr && \
+make && make install && \
+rm -rf /apr-util-1.5.4.tar.gz /apr-util-1.5.4
+
+# Libtool
+RUN curl -L -O https://ftp.gnu.org/gnu/libtool/libtool-2.4.6.tar.gz && \
+tar xvfz libtool-2.4.6.tar.gz && \
+cd 

[GitHub] merlimat closed pull request #1793: Added docker files and script to build static client library and Python bindings on Linux

[GitBox]

merlimat closed pull request #1793: Added docker files and script to build 
static client library and Python bindings on Linux
URL: https://github.com/apache/incubator-pulsar/pull/1793
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pulsar-client-cpp/docker/Dockerfile 
b/pulsar-client-cpp/docker/Dockerfile
new file mode 100644
index 00..1df9545695
--- /dev/null
+++ b/pulsar-client-cpp/docker/Dockerfile
@@ -0,0 +1,144 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+
+FROM quay.io/pypa/manylinux1_x86_64
+
+RUN yum install -y gtest-devel
+
+ARG PYTHON_VERSION
+ARG PYTHON_SPEC
+
+ENV PYTHON_VERSION=${PYTHON_VERSION}
+ENV PYTHON_SPEC=${PYTHON_SPEC}
+
+ENV PATH="/opt/python/${PYTHON_SPEC}/bin:${PATH}"
+
+RUN ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m 
/opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}
+
+
+# These dependencies can be found in Ubuntu but they're not compiled with 
-fPIC,
+# so they cannot be statically linked into a shared library
+
+
+# ZLib
+RUN curl -O -L https://zlib.net/zlib-1.2.11.tar.gz && \
+tar xvfz zlib-1.2.11.tar.gz && \
+cd zlib-1.2.11 && \
+CFLAGS="-fPIC -O3" ./configure && \
+make && make install && \
+rm -rf /zlib-1.2.11.tar.gz /zlib-1.2.11
+
+# Compile OpenSSL
+RUN curl -O -L 
https://github.com/openssl/openssl/archive/OpenSSL_1_0_2l.tar.gz && \
+tar xvfz OpenSSL_1_0_2l.tar.gz && \
+cd openssl-OpenSSL_1_0_2l/ && \
+./Configure -fPIC no-shared linux-x86_64 && \
+make && make install && \
+rm -rf /OpenSSL_1_0_2l.tar.gz /openssl-OpenSSL_1_0_2l
+
+# Download and compile boost
+RUN curl -O -L 
https://dl.bintray.com/boostorg/release/1.64.0/source/boost_1_64_0.tar.gz && \
+tar xvfz boost_1_64_0.tar.gz && \
+cd /boost_1_64_0 && \
+./bootstrap.sh 
--with-libraries=program_options,filesystem,regex,thread,system,python && \
+./b2 address-model=64 cxxflags=-fPIC link=static threading=multi 
variant=release install && \
+rm -rf /boost_1_64_0.tar.gz /boost_1_64_0
+
+# Download and copile protoubf
+RUN curl -O -L  
https://github.com/google/protobuf/releases/download/v3.3.0/protobuf-cpp-3.3.0.tar.gz
 && \
+tar xvfz protobuf-cpp-3.3.0.tar.gz && \
+cd protobuf-3.3.0/ && \
+CXXFLAGS=-fPIC ./configure && \
+make && make install && ldconfig && \
+rm -rf /protobuf-cpp-3.3.0.tar.gz /protobuf-3.3.0
+
+# Compile APR
+RUN curl -O -L  http://archive.apache.org/dist/apr/apr-1.5.2.tar.gz && \
+tar xvfz apr-1.5.2.tar.gz && \
+cd apr-1.5.2 && \
+CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure && \
+make && make install && \
+rm -rf /apr-1.5.2.tar.gz /apr-1.5.2
+
+# Compile APR-Util
+RUN curl -O -L  http://archive.apache.org/dist/apr/apr-util-1.5.4.tar.gz && \
+tar xvfz apr-util-1.5.4.tar.gz && \
+cd apr-util-1.5.4 && \
+CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure -with-apr=/usr/local/apr && \
+make && make install && \
+rm -rf /apr-util-1.5.4.tar.gz /apr-util-1.5.4
+
+# Libtool
+RUN curl -L -O https://ftp.gnu.org/gnu/libtool/libtool-2.4.6.tar.gz && \
+tar xvfz libtool-2.4.6.tar.gz && \
+cd libtool-2.4.6 && \
+./configure && \
+make && make install && \
+rm -rf /libtool-2.4.6.tar.gz /libtool-2.4.6
+
+# Compile log4cxx
+RUN git clone https://github.com/apache/logging-log4cxx.git && \
+cd logging-log4cxx && \
+./autogen.sh && \
+CXXFLAGS=-fPIC ./configure && \
+make && make install && \
+rm -rf /logging-log4cxx
+
+# Compile expat
+RUN curl -O -L  https://github.com/libexpat/libexpat/archive/R_2_2_0.tar.gz && 
\
+tar xfvz R_2_2_0.tar.gz && \
+cd libexpat-R_2_2_0/expat && \
+./buildconf.sh && \
+CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure  && \
+make && make installlib && \
+rm -rf /R_2_2_0.tar.gz /libexpat-R_2_2_0
+
+RUN curl -O -L https://github.com/Kitware/CMake/archive/v3.8.2.tar.gz && \
+tar xvfz v3.8.2.tar.gz && \
+cd CMake-3.8.2 && \
+   

[GitHub] lucperkins commented on issue #1763: Document Python dependencies

[GitBox]

lucperkins commented on issue #1763: Document Python dependencies
URL: https://github.com/apache/incubator-pulsar/pull/1763#issuecomment-390023082
 
 
   @merlimat Someday I'll remember on my own grr. Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on a change in pull request #1799: BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength

[GitBox]

ivankelly commented on a change in pull request #1799: 
BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength
URL: https://github.com/apache/incubator-pulsar/pull/1799#discussion_r189098223
 
 

 ##
 File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
 ##
 @@ -122,14 +122,15 @@ private int readEntries() throws IOException {
 Iterator iterator = ledgerEntriesOnce.iterator();
 while (iterator.hasNext()) {
 LedgerEntry entry = iterator.next();
-int entryLength = (int) entry.getLength();
+ByteBuf buf = entry.getEntryBuffer().retain();
 
 Review comment:
   The refCount for this retain has its ownership transferred to the composite 
bytebuf created below, so it will get released when that bytebuf is released.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1763: Document Python dependencies

[GitBox]

merlimat commented on issue #1763: Document Python dependencies
URL: https://github.com/apache/incubator-pulsar/pull/1763#issuecomment-390005862
 
 
   @lucperkins There are some missing headers: 
   
   ```
   2018-05-17\T\16:27:06.785 [WARNING] Missing header in: 
/home/jenkins/jenkins-slave/workspace/pulsar_precommit_java8/site/_data/deps.yaml
   2018-05-17\T\16:27:06.799 [WARNING] Missing header in: 
/home/jenkins/jenkins-slave/workspace/pulsar_precommit_java8/site/_includes/python-deps.html
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1793: Added docker files and script to build static client library and Python bindings on Linux

[GitBox]

merlimat commented on issue #1793: Added docker files and script to build 
static client library and Python bindings on Linux
URL: https://github.com/apache/incubator-pulsar/pull/1793#issuecomment-389949461
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Remove old comment (#1803)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new f09eb3c  Remove old comment (#1803)
f09eb3c is described below

commit f09eb3cdbd8542c9f19efbb281891aa0c435116b
Author: Boyang Jerry Peng 
AuthorDate: Thu May 17 12:59:55 2018 -0700

Remove old comment (#1803)
---
 .../core/src/main/java/org/apache/pulsar/io/core/Sink.java | 10 +-
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java 
b/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java
index 0265e77..a84a47e 100644
--- a/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java
+++ b/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java
@@ -31,15 +31,7 @@ public interface Sink extends AutoCloseable{
  * @throws Exception IO type exceptions when opening a connector
  */
 void open(final Map config) throws Exception;
-
-/**
- * Attempt to publish a type safe collection of messages
- *
- * @param value output value
- * @return Completable future fo async publish request
- */
-
-
+
 /**
  * Write a message to Sink
  * @param inputRecordContext Context of value

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[GitHub] merlimat closed pull request #1803: Remove old comment

[GitBox]

merlimat closed pull request #1803: Remove old comment
URL: https://github.com/apache/incubator-pulsar/pull/1803
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java 
b/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java
index 0265e77ce8..a84a47e071 100644
--- a/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java
+++ b/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java
@@ -31,15 +31,7 @@
  * @throws Exception IO type exceptions when opening a connector
  */
 void open(final Map config) throws Exception;
-
-/**
- * Attempt to publish a type safe collection of messages
- *
- * @param value output value
- * @return Completable future fo async publish request
- */
-
-
+
 /**
  * Write a message to Sink
  * @param inputRecordContext Context of value


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jerrypeng opened a new pull request #1803: Remove old comment

[GitBox]

jerrypeng opened a new pull request #1803: Remove old comment
URL: https://github.com/apache/incubator-pulsar/pull/1803
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1802: Utilities to update configuration for integration tests

[GitBox]

merlimat closed pull request #1802: Utilities to update configuration for 
integration tests
URL: https://github.com/apache/incubator-pulsar/pull/1802
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/tests/integration-tests-utils/src/main/java/org/apache/pulsar/tests/PulsarClusterUtils.java
 
b/tests/integration-tests-utils/src/main/java/org/apache/pulsar/tests/PulsarClusterUtils.java
index 984dadc1d9..ca8e33b0d7 100644
--- 
a/tests/integration-tests-utils/src/main/java/org/apache/pulsar/tests/PulsarClusterUtils.java
+++ 
b/tests/integration-tests-utils/src/main/java/org/apache/pulsar/tests/PulsarClusterUtils.java
@@ -169,7 +169,7 @@ public static boolean waitBrokerUp(DockerClient docker, 
String containerId,
 return true;
 } catch (Exception e) {
 // couldn't connect, try again after sleep
-LOG.info("Failed to connect {} @ {}", ip, BROKER_PORT, e);
+LOG.debug("Failed to connect {} @ {}", ip, BROKER_PORT, e);
 }
 try {
 Thread.sleep(pollMillis);
@@ -295,4 +295,27 @@ public static void stopAllProxies(DockerClient docker, 
String cluster) {
 return DockerUtils.cubeIdsWithLabels(docker, 
ImmutableMap.of("service", "zookeeper",
  
"cluster", cluster));
 }
+
+public static void updateConf(DockerClient docker, String containerId,
+  String confFile, String key, String value) 
throws Exception {
+String sedProgram = String.format(
+"/[[:blank:]]*%s[[:blank:]]*=/ { h; s^=.*^=%s^; }; ${x;/^$/ { 
s^^%s=%s^;H; }; x}",
+key, value, key, value);
+DockerUtils.runCommand(docker, containerId, "sed", "-i", "-e", 
sedProgram, confFile);
+}
+
+public static void setLogLevel(DockerClient docker, String containerId,
+   String loggerName, String level) throws 
Exception {
+String sedProgram = String.format(
+"/  Logger:/ a\\\n"
++"  - name: %s\\n"
++"level: %s\\n"
++"additivity: false\\n"
++"AppenderRef:\\n"
++"  - ref: Console\\n"
++"  - level: debug\\n",
+loggerName, level);
+String logConf = "/pulsar/conf/log4j2.yaml";
+DockerUtils.runCommand(docker, containerId, "sed", "-i", "-e", 
sedProgram, logConf);
+}
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Utilities to update configuration for integration tests (#1802)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 63a6b83  Utilities to update configuration for integration tests 
(#1802)
63a6b83 is described below

commit 63a6b835bb3f21b63923eb128c285b9f827491d0
Author: Ivan Kelly 
AuthorDate: Thu May 17 19:43:35 2018 +0200

Utilities to update configuration for integration tests (#1802)

This patch provides two utility methods. One of them allows the update
of a key value properties type configuration, such as broker.conf.

The other allows the update of a log4j2.yaml, to add loggers at a certain
level. This is useful for debugging integration tests.
---
 .../apache/pulsar/tests/PulsarClusterUtils.java| 25 +-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git 
a/tests/integration-tests-utils/src/main/java/org/apache/pulsar/tests/PulsarClusterUtils.java
 
b/tests/integration-tests-utils/src/main/java/org/apache/pulsar/tests/PulsarClusterUtils.java
index 984dadc..ca8e33b 100644
--- 
a/tests/integration-tests-utils/src/main/java/org/apache/pulsar/tests/PulsarClusterUtils.java
+++ 
b/tests/integration-tests-utils/src/main/java/org/apache/pulsar/tests/PulsarClusterUtils.java
@@ -169,7 +169,7 @@ public class PulsarClusterUtils {
 return true;
 } catch (Exception e) {
 // couldn't connect, try again after sleep
-LOG.info("Failed to connect {} @ {}", ip, BROKER_PORT, e);
+LOG.debug("Failed to connect {} @ {}", ip, BROKER_PORT, e);
 }
 try {
 Thread.sleep(pollMillis);
@@ -295,4 +295,27 @@ public class PulsarClusterUtils {
 return DockerUtils.cubeIdsWithLabels(docker, 
ImmutableMap.of("service", "zookeeper",
  
"cluster", cluster));
 }
+
+public static void updateConf(DockerClient docker, String containerId,
+  String confFile, String key, String value) 
throws Exception {
+String sedProgram = String.format(
+"/[[:blank:]]*%s[[:blank:]]*=/ { h; s^=.*^=%s^; }; ${x;/^$/ { 
s^^%s=%s^;H; }; x}",
+key, value, key, value);
+DockerUtils.runCommand(docker, containerId, "sed", "-i", "-e", 
sedProgram, confFile);
+}
+
+public static void setLogLevel(DockerClient docker, String containerId,
+   String loggerName, String level) throws 
Exception {
+String sedProgram = String.format(
+"/  Logger:/ a\\\n"
++"  - name: %s\\n"
++"level: %s\\n"
++"additivity: false\\n"
++"AppenderRef:\\n"
++"  - ref: Console\\n"
++"  - level: debug\\n",
+loggerName, level);
+String logConf = "/pulsar/conf/log4j2.yaml";
+DockerUtils.runCommand(docker, containerId, "sed", "-i", "-e", 
sedProgram, logConf);
+}
 }

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[incubator-pulsar] branch master updated: Log4j2.yaml configures loggers incorrectly (#1801)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new e22ef77  Log4j2.yaml configures loggers incorrectly (#1801)
e22ef77 is described below

commit e22ef77c35c9412461af16113b8107d8933959b4
Author: Ivan Kelly 
AuthorDate: Thu May 17 19:40:41 2018 +0200

Log4j2.yaml configures loggers incorrectly (#1801)

log4j2.yaml has a list of loggers. However, this is not a yaml list,
but a yaml map, so the configuration only picks up the last one.

This patch changes the loggers to a yaml list.
---
 conf/log4j2.yaml | 48 +++-
 1 file changed, 23 insertions(+), 25 deletions(-)

diff --git a/conf/log4j2.yaml b/conf/log4j2.yaml
index cd0ec88..d98cd49 100644
--- a/conf/log4j2.yaml
+++ b/conf/log4j2.yaml
@@ -156,20 +156,6 @@ Configuration:
 
   Loggers:
 
-Logger:
-  name: org.apache.bookkeeper
-  level: "${sys:bk.log.level}"
-  additivity: false
-  AppenderRef:
-- ref: BkRollingFile
-
-Logger:
-  name: org.apache.distributedlog
-  level: "${sys:bk.log.level}"
-  additivity: false
-  AppenderRef:
-- ref: BkRollingFile
-  
 # Default root logger configuration
 Root:
   level: info
@@ -177,17 +163,29 @@ Configuration:
   AppenderRef:
 - ref: "${sys:pulsar.log.appender}"
   level: "${sys:pulsar.log.level}"
+
+Logger:
+  - name: org.apache.bookkeeper
+level: "${sys:bk.log.level}"
+additivity: false
+AppenderRef:
+  - ref: BkRollingFile
+
+  - name: org.apache.distributedlog
+level: "${sys:bk.log.level}"
+additivity: false
+AppenderRef:
+  - ref: BkRollingFile
 
 # Logger to inject filter script
-#Logger:
-#  name: org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl
-#  level: debug
-#  additivity: false
-#  AppenderRef:
-#ref: "${sys:pulsar.log.appender}"
-#ScriptFilter:
-#  onMatch: ACCEPT
-#  onMisMatch: DENY
-#  ScriptRef:
-#ref: filter.js
+# - name: org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl
+#   level: debug
+#   additivity: false
+#   AppenderRef:
+# ref: "${sys:pulsar.log.appender}"
+# ScriptFilter:
+#   onMatch: ACCEPT
+#   onMisMatch: DENY
+#   ScriptRef:
+# ref: filter.js
 

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[GitHub] merlimat closed pull request #1801: Log4j2.yaml configures loggers incorrectly

[GitBox]

merlimat closed pull request #1801: Log4j2.yaml configures loggers incorrectly
URL: https://github.com/apache/incubator-pulsar/pull/1801
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/conf/log4j2.yaml b/conf/log4j2.yaml
index cd0ec880b6..d98cd49220 100644
--- a/conf/log4j2.yaml
+++ b/conf/log4j2.yaml
@@ -156,20 +156,6 @@ Configuration:
 
   Loggers:
 
-Logger:
-  name: org.apache.bookkeeper
-  level: "${sys:bk.log.level}"
-  additivity: false
-  AppenderRef:
-- ref: BkRollingFile
-
-Logger:
-  name: org.apache.distributedlog
-  level: "${sys:bk.log.level}"
-  additivity: false
-  AppenderRef:
-- ref: BkRollingFile
-  
 # Default root logger configuration
 Root:
   level: info
@@ -177,17 +163,29 @@ Configuration:
   AppenderRef:
 - ref: "${sys:pulsar.log.appender}"
   level: "${sys:pulsar.log.level}"
+
+Logger:
+  - name: org.apache.bookkeeper
+level: "${sys:bk.log.level}"
+additivity: false
+AppenderRef:
+  - ref: BkRollingFile
+
+  - name: org.apache.distributedlog
+level: "${sys:bk.log.level}"
+additivity: false
+AppenderRef:
+  - ref: BkRollingFile
 
 # Logger to inject filter script
-#Logger:
-#  name: org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl
-#  level: debug
-#  additivity: false
-#  AppenderRef:
-#ref: "${sys:pulsar.log.appender}"
-#ScriptFilter:
-#  onMatch: ACCEPT
-#  onMisMatch: DENY
-#  ScriptRef:
-#ref: filter.js
+# - name: org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl
+#   level: debug
+#   additivity: false
+#   AppenderRef:
+# ref: "${sys:pulsar.log.appender}"
+# ScriptFilter:
+#   onMatch: ACCEPT
+#   onMisMatch: DENY
+#   ScriptRef:
+# ref: filter.js
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1793: Added docker files and script to build static client library and Python bindings on Linux

[GitBox]

merlimat commented on issue #1793: Added docker files and script to build 
static client library and Python bindings on Linux
URL: https://github.com/apache/incubator-pulsar/pull/1793#issuecomment-389949461
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1799: BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength

[GitBox]

merlimat commented on a change in pull request #1799: 
BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength
URL: https://github.com/apache/incubator-pulsar/pull/1799#discussion_r189042435
 
 

 ##
 File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
 ##
 @@ -122,14 +122,15 @@ private int readEntries() throws IOException {
 Iterator iterator = ledgerEntriesOnce.iterator();
 while (iterator.hasNext()) {
 LedgerEntry entry = iterator.next();
-int entryLength = (int) entry.getLength();
+ByteBuf buf = entry.getEntryBuffer().retain();
 
 Review comment:
   Shouldn't this retain be compensated with a `release()`. I assume the 
entries returned by this method will be released but the entryBuffer itself 
would have to be released.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Modifying sink interface to be generic (#1792)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 5e4f2bb  Modifying sink interface to be generic (#1792)
5e4f2bb is described below

commit 5e4f2bbbf791e216e2eb6c50088c1c5910f0ec20
Author: Boyang Jerry Peng 
AuthorDate: Thu May 17 10:25:24 2018 -0700

Modifying sink interface to be generic (#1792)
---
 .../functions/instance/JavaInstanceRunnable.java   | 10 +--
 .../apache/pulsar/functions/sink/PulsarSink.java   |  8 +-
 .../apache/pulsar/functions/sink/RuntimeSink.java  | 51 
 .../functions/sink/DefaultRuntimeSinkTest.java | 96 --
 .../apache/pulsar/io/aerospike/AerospikeSink.java  |  3 +-
 .../apache/pulsar/io/cassandra/CassandraSink.java  |  3 +-
 .../java/org/apache/pulsar/io/core/SimpleSink.java | 46 +++
 .../main/java/org/apache/pulsar/io/core/Sink.java  | 21 ++---
 .../java/org/apache/pulsar/io/kafka/KafkaSink.java |  3 +-
 9 files changed, 34 insertions(+), 207 deletions(-)

diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
index 4aaed5b..5b5e943 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
@@ -56,10 +56,8 @@ import org.apache.pulsar.functions.api.Function;
 import org.apache.pulsar.functions.proto.InstanceCommunication;
 import org.apache.pulsar.functions.proto.Function.SourceSpec;
 import org.apache.pulsar.functions.proto.Function.SinkSpec;
-import org.apache.pulsar.functions.sink.DefaultRuntimeSink;
 import org.apache.pulsar.functions.sink.PulsarSink;
 import org.apache.pulsar.functions.sink.PulsarSinkConfig;
-import org.apache.pulsar.functions.sink.RuntimeSink;
 import org.apache.pulsar.functions.source.PulsarRecord;
 import org.apache.pulsar.functions.source.PulsarSource;
 import org.apache.pulsar.functions.source.PulsarSourceConfig;
@@ -106,7 +104,7 @@ public class JavaInstanceRunnable implements AutoCloseable, 
Runnable {
 private Record currentRecord;
 
 private Source source;
-private RuntimeSink sink;
+private Sink sink;
 
 public JavaInstanceRunnable(InstanceConfig instanceConfig,
 FunctionCacheManager fnCache,
@@ -524,10 +522,8 @@ public class JavaInstanceRunnable implements 
AutoCloseable, Runnable {
 Thread.currentThread().getContextClassLoader());
 }
 
-if (object instanceof RuntimeSink) {
-this.sink = (RuntimeSink) object;
-} else if (object instanceof Sink) {
-this.sink = DefaultRuntimeSink.of((Sink) object);
+if (object instanceof Sink) {
+this.sink = (Sink) object;
 } else {
 throw new RuntimeException("Sink does not implement correct 
interface");
 }
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
index 61deeff..4fccb54 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
@@ -39,13 +39,14 @@ import 
org.apache.pulsar.functions.instance.producers.Producers;
 import org.apache.pulsar.functions.source.PulsarRecord;
 import org.apache.pulsar.functions.utils.FunctionConfig;
 import org.apache.pulsar.io.core.RecordContext;
+import org.apache.pulsar.io.core.Sink;
 
 import java.util.Base64;
 import java.util.Map;
 import java.util.concurrent.CompletableFuture;
 
 @Slf4j
-public class PulsarSink implements RuntimeSink {
+public class PulsarSink implements Sink {
 
 private PulsarClient client;
 private PulsarSinkConfig pulsarSinkConfig;
@@ -207,11 +208,6 @@ public class PulsarSink implements RuntimeSink {
 }
 
 @Override
-public CompletableFuture write(T value) {
-return null;
-}
-
-@Override
 public void write(RecordContext recordContext, T value) throws Exception {
 
 byte[] output;
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/RuntimeSink.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/RuntimeSink.java
deleted file mode 100644
index e9c8dc5..000
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/RuntimeSink.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license 

[GitHub] merlimat closed pull request #1792: Modifying sink interface to be generic

[GitBox]

merlimat closed pull request #1792: Modifying sink interface to be generic
URL: https://github.com/apache/incubator-pulsar/pull/1792
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
index 4aaed5bab3..5b5e943e84 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
@@ -56,10 +56,8 @@
 import org.apache.pulsar.functions.proto.InstanceCommunication;
 import org.apache.pulsar.functions.proto.Function.SourceSpec;
 import org.apache.pulsar.functions.proto.Function.SinkSpec;
-import org.apache.pulsar.functions.sink.DefaultRuntimeSink;
 import org.apache.pulsar.functions.sink.PulsarSink;
 import org.apache.pulsar.functions.sink.PulsarSinkConfig;
-import org.apache.pulsar.functions.sink.RuntimeSink;
 import org.apache.pulsar.functions.source.PulsarRecord;
 import org.apache.pulsar.functions.source.PulsarSource;
 import org.apache.pulsar.functions.source.PulsarSourceConfig;
@@ -106,7 +104,7 @@
 private Record currentRecord;
 
 private Source source;
-private RuntimeSink sink;
+private Sink sink;
 
 public JavaInstanceRunnable(InstanceConfig instanceConfig,
 FunctionCacheManager fnCache,
@@ -524,10 +522,8 @@ public void setupOutput() throws Exception {
 Thread.currentThread().getContextClassLoader());
 }
 
-if (object instanceof RuntimeSink) {
-this.sink = (RuntimeSink) object;
-} else if (object instanceof Sink) {
-this.sink = DefaultRuntimeSink.of((Sink) object);
+if (object instanceof Sink) {
+this.sink = (Sink) object;
 } else {
 throw new RuntimeException("Sink does not implement correct 
interface");
 }
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
index 61deeff9e4..4fccb54451 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
@@ -39,13 +39,14 @@
 import org.apache.pulsar.functions.source.PulsarRecord;
 import org.apache.pulsar.functions.utils.FunctionConfig;
 import org.apache.pulsar.io.core.RecordContext;
+import org.apache.pulsar.io.core.Sink;
 
 import java.util.Base64;
 import java.util.Map;
 import java.util.concurrent.CompletableFuture;
 
 @Slf4j
-public class PulsarSink implements RuntimeSink {
+public class PulsarSink implements Sink {
 
 private PulsarClient client;
 private PulsarSinkConfig pulsarSinkConfig;
@@ -206,11 +207,6 @@ public void open(Map config) throws 
Exception {
 
this.pulsarSinkProcessor.initializeOutputProducer(this.pulsarSinkConfig.getTopic());
 }
 
-@Override
-public CompletableFuture write(T value) {
-return null;
-}
-
 @Override
 public void write(RecordContext recordContext, T value) throws Exception {
 
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/RuntimeSink.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/RuntimeSink.java
deleted file mode 100644
index e9c8dc5b43..00
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/RuntimeSink.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *   http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pulsar.functions.sink;
-
-import org.apache.pulsar.io.core.RecordContext;
-import org.apache.pulsar.io.core.Sink;
-
-/**
- * This class 

[GitHub] ivankelly opened a new pull request #1802: Utilities to update configuration for integration tests

[GitBox]

ivankelly opened a new pull request #1802: Utilities to update configuration 
for integration tests
URL: https://github.com/apache/incubator-pulsar/pull/1802
 
 
   This patch provides two utility methods. One of them allows the update
   of a key value properties type configuration, such as broker.conf.
   
   The other allows the update of a log4j2.yaml, to add loggers at a certain
   level. This is useful for debugging integration tests.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #1801: Log4j2.yaml configures loggers incorrectly

[GitBox]

ivankelly opened a new pull request #1801: Log4j2.yaml configures loggers 
incorrectly
URL: https://github.com/apache/incubator-pulsar/pull/1801
 
 
   log4j2.yaml has a list of loggers. However, this is not a yaml list,
   but a yaml map, so the configuration only picks up the last one.
   
   This patch changes the loggers to a yaml list.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #1800: Docker testing init script has wrong service url

[GitBox]

ivankelly opened a new pull request #1800: Docker testing init script has wrong 
service url
URL: https://github.com/apache/incubator-pulsar/pull/1800
 
 
   The script was initializing the broker server url with a http url,
   using the binary protocol port. This patch fixes that and also logs
   the bash commands to make this easier to debug in future.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #1798: S3ManagedLedgerOffloader should require region or endpoint

[GitBox]

ivankelly opened a new pull request #1798: S3ManagedLedgerOffloader should 
require region or endpoint
URL: https://github.com/apache/incubator-pulsar/pull/1798
 
 
   The S3 client requires one of the two, but never both. Previously we
   were throwing an error if region was not specified, even though it's
   not needed if endpoint is.
   
   Master Issue: #1511
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #1799: BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength

[GitBox]

ivankelly opened a new pull request #1799: BlockAwareSegmentInputStreamImpl 
shouldn't use LedgerEntry#getLength
URL: https://github.com/apache/incubator-pulsar/pull/1799
 
 
   LedgerEntry#getLength() returns the length of the ledger up to and
   including the length of the entry, not the length of the entry as you
   would expected (and the documentation states). So we shouldn't use it
   when serializing the entry, because what we need is the length of the
   entry.
   
   This patch changes BlockAwareSegmentInputStreamImpl to use the
   readable bytes of the bytebuf instead.
   
   Master Issue: #1511
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1795: Fix C++ MurmurHash3 algorithm

[GitBox]

merlimat closed pull request #1795: Fix C++ MurmurHash3 algorithm
URL: https://github.com/apache/incubator-pulsar/pull/1795
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pulsar-client-cpp/lib/Murmur3_32Hash.cc 
b/pulsar-client-cpp/lib/Murmur3_32Hash.cc
index ef88590560..d45169c3f8 100644
--- a/pulsar-client-cpp/lib/Murmur3_32Hash.cc
+++ b/pulsar-client-cpp/lib/Murmur3_32Hash.cc
@@ -70,14 +70,15 @@ uint32_t Murmur3_32Hash::makeHash(const void *key, const 
int64_t len) {
 h1 = mixH1(h1, k1);
 }
 
+const uint8_t *tail = reinterpret_cast(data + nblocks * 
MACRO_CHUNK_SIZE);
 uint32_t k1 = 0;
 switch (len - nblocks * MACRO_CHUNK_SIZE) {
 case 3:
-k1 ^= static_cast(blocks[2]) << 16;
+k1 ^= static_cast(tail[2]) << 16;
 case 2:
-k1 ^= static_cast(blocks[1]) << 8;
+k1 ^= static_cast(tail[1]) << 8;
 case 1:
-k1 ^= static_cast(blocks[0]);
+k1 ^= static_cast(tail[0]);
 };
 
 h1 ^= mixK1(k1);
diff --git a/pulsar-client-cpp/tests/HashTest.cc 
b/pulsar-client-cpp/tests/HashTest.cc
index 97dcb90501..bd6de09ed6 100644
--- a/pulsar-client-cpp/tests/HashTest.cc
+++ b/pulsar-client-cpp/tests/HashTest.cc
@@ -58,10 +58,18 @@ TEST(HashTest, testJavaStringHash) {
 
 TEST(HashTest, testMurmur3_32Hash) {
 Murmur3_32Hash hash;
+std::string k1 = "k1";
+std::string k2 = "k2";
 std::string key1 = "key1";
 std::string key2 = "key2";
+std::string key01 = "key01";
+std::string key02 = "key02";
 
 // Same value as Java client
+ASSERT_EQ(2110152746, hash.makeHash(k1));
+ASSERT_EQ(147994, hash.makeHash(k2));
 ASSERT_EQ(462881061, hash.makeHash(key1));
 ASSERT_EQ(1936800180, hash.makeHash(key2));
-}
\ No newline at end of file
+ASSERT_EQ(39696932, hash.makeHash(key01));
+ASSERT_EQ(751761803, hash.makeHash(key02));
+}
diff --git 
a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java 
b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java
index f53205d47e..c10c8d0c7a 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java
@@ -43,7 +43,11 @@ public void murmur3_32HashTest() {
 Hash h = Murmur3_32Hash.getInstance();
 
 // Same value as C++ client
+assertEquals(2110152746, h.makeHash("k1"));
+assertEquals(147994, h.makeHash("k2"));
 assertEquals(462881061, h.makeHash("key1"));
 assertEquals(1936800180, h.makeHash("key2"));
+assertEquals(39696932, h.makeHash("key01"));
+assertEquals(751761803, h.makeHash("key02"));
 }
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Fix C++ MurmurHash3 algorithm (#1795)

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 920ecba  Fix C++ MurmurHash3 algorithm (#1795)
920ecba is described below

commit 920ecba6fc53c81191fac011212eccac48186e88
Author: massakam 
AuthorDate: Fri May 18 01:07:15 2018 +0900

Fix C++ MurmurHash3 algorithm (#1795)

* Fix C++ MurmurHash3 algorithm

* Add the same test cases to HashTest.java
---
 pulsar-client-cpp/lib/Murmur3_32Hash.cc|  7 ---
 pulsar-client-cpp/tests/HashTest.cc| 10 +-
 .../src/test/java/org/apache/pulsar/client/impl/HashTest.java  |  4 
 3 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/pulsar-client-cpp/lib/Murmur3_32Hash.cc 
b/pulsar-client-cpp/lib/Murmur3_32Hash.cc
index ef88590..d45169c 100644
--- a/pulsar-client-cpp/lib/Murmur3_32Hash.cc
+++ b/pulsar-client-cpp/lib/Murmur3_32Hash.cc
@@ -70,14 +70,15 @@ uint32_t Murmur3_32Hash::makeHash(const void *key, const 
int64_t len) {
 h1 = mixH1(h1, k1);
 }
 
+const uint8_t *tail = reinterpret_cast(data + nblocks * 
MACRO_CHUNK_SIZE);
 uint32_t k1 = 0;
 switch (len - nblocks * MACRO_CHUNK_SIZE) {
 case 3:
-k1 ^= static_cast(blocks[2]) << 16;
+k1 ^= static_cast(tail[2]) << 16;
 case 2:
-k1 ^= static_cast(blocks[1]) << 8;
+k1 ^= static_cast(tail[1]) << 8;
 case 1:
-k1 ^= static_cast(blocks[0]);
+k1 ^= static_cast(tail[0]);
 };
 
 h1 ^= mixK1(k1);
diff --git a/pulsar-client-cpp/tests/HashTest.cc 
b/pulsar-client-cpp/tests/HashTest.cc
index 97dcb90..bd6de09 100644
--- a/pulsar-client-cpp/tests/HashTest.cc
+++ b/pulsar-client-cpp/tests/HashTest.cc
@@ -58,10 +58,18 @@ TEST(HashTest, testJavaStringHash) {
 
 TEST(HashTest, testMurmur3_32Hash) {
 Murmur3_32Hash hash;
+std::string k1 = "k1";
+std::string k2 = "k2";
 std::string key1 = "key1";
 std::string key2 = "key2";
+std::string key01 = "key01";
+std::string key02 = "key02";
 
 // Same value as Java client
+ASSERT_EQ(2110152746, hash.makeHash(k1));
+ASSERT_EQ(147994, hash.makeHash(k2));
 ASSERT_EQ(462881061, hash.makeHash(key1));
 ASSERT_EQ(1936800180, hash.makeHash(key2));
-}
\ No newline at end of file
+ASSERT_EQ(39696932, hash.makeHash(key01));
+ASSERT_EQ(751761803, hash.makeHash(key02));
+}
diff --git 
a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java 
b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java
index f53205d..c10c8d0 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java
@@ -43,7 +43,11 @@ public class HashTest {
 Hash h = Murmur3_32Hash.getInstance();
 
 // Same value as C++ client
+assertEquals(2110152746, h.makeHash("k1"));
+assertEquals(147994, h.makeHash("k2"));
 assertEquals(462881061, h.makeHash("key1"));
 assertEquals(1936800180, h.makeHash("key2"));
+assertEquals(39696932, h.makeHash("key01"));
+assertEquals(751761803, h.makeHash("key02"));
 }
 }

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[incubator-pulsar] branch master updated: adding windowing CLI option comments (#1788)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 177e88d  adding windowing CLI option comments (#1788)
177e88d is described below

commit 177e88d97e0f7a5745189c0640e2e6192f76c0bc
Author: Boyang Jerry Peng 
AuthorDate: Thu May 17 08:50:09 2018 -0700

adding windowing CLI option comments (#1788)
---
 .../src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java   | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
index dc311ba..30103c7 100644
--- 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
+++ 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
@@ -231,13 +231,13 @@ public class CmdFunctions extends CmdBase {
 protected Long ram;
 @Parameter(names = "--disk", description = "The disk in bytes that 
need to be allocated per function instance(applicable only to docker runtime)")
 protected Long disk;
-@Parameter(names = "--windowLengthCount", description = "")
+@Parameter(names = "--windowLengthCount", description = "The number of 
messages per window")
 protected Integer windowLengthCount;
-@Parameter(names = "--windowLengthDurationMs", description = "")
+@Parameter(names = "--windowLengthDurationMs", description = "The time 
duration of the window in milliseconds")
 protected Long windowLengthDurationMs;
-@Parameter(names = "--slidingIntervalCount", description = "")
+@Parameter(names = "--slidingIntervalCount", description = "The number 
of messages after which the window slides")
 protected Integer slidingIntervalCount;
-@Parameter(names = "--slidingIntervalDurationMs", description = "")
+@Parameter(names = "--slidingIntervalDurationMs", description = "The 
time duration after which the window slides")
 protected Long slidingIntervalDurationMs;
 @Parameter(names = "--autoAck", description = "")
 protected Boolean autoAck;

-- 
To stop receiving notification emails like this one, please contact
si...@apache.org.

[GitHub] sijie closed pull request #1788: adding windowing function CLI option descriptions

[GitBox]

sijie closed pull request #1788: adding windowing function CLI option 
descriptions
URL: https://github.com/apache/incubator-pulsar/pull/1788
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
index dc311bafaa..30103c7360 100644
--- 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
+++ 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
@@ -231,13 +231,13 @@ void processArguments() throws Exception {
 protected Long ram;
 @Parameter(names = "--disk", description = "The disk in bytes that 
need to be allocated per function instance(applicable only to docker runtime)")
 protected Long disk;
-@Parameter(names = "--windowLengthCount", description = "")
+@Parameter(names = "--windowLengthCount", description = "The number of 
messages per window")
 protected Integer windowLengthCount;
-@Parameter(names = "--windowLengthDurationMs", description = "")
+@Parameter(names = "--windowLengthDurationMs", description = "The time 
duration of the window in milliseconds")
 protected Long windowLengthDurationMs;
-@Parameter(names = "--slidingIntervalCount", description = "")
+@Parameter(names = "--slidingIntervalCount", description = "The number 
of messages after which the window slides")
 protected Integer slidingIntervalCount;
-@Parameter(names = "--slidingIntervalDurationMs", description = "")
+@Parameter(names = "--slidingIntervalDurationMs", description = "The 
time duration after which the window slides")
 protected Long slidingIntervalDurationMs;
 @Parameter(names = "--autoAck", description = "")
 protected Boolean autoAck;


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1795: Fix C++ MurmurHash3 algorithm

[GitBox]

sijie commented on issue #1795: Fix C++ MurmurHash3 algorithm
URL: https://github.com/apache/incubator-pulsar/pull/1795#issuecomment-389914752
 
 
   we might consider porting this change back for 2.0.1?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nkurihar commented on a change in pull request #1795: Fix C++ MurmurHash3 algorithm

[GitBox]

nkurihar commented on a change in pull request #1795: Fix C++ MurmurHash3 
algorithm
URL: https://github.com/apache/incubator-pulsar/pull/1795#discussion_r188907542
 
 

 ##
 File path: pulsar-client-cpp/tests/HashTest.cc
 ##
 @@ -58,10 +58,18 @@ TEST(HashTest, testJavaStringHash) {
 
 TEST(HashTest, testMurmur3_32Hash) {
 Murmur3_32Hash hash;
+std::string k1 = "k1";
+std::string k2 = "k2";
 std::string key1 = "key1";
 std::string key2 = "key2";
+std::string key01 = "key01";
+std::string key02 = "key02";
 
 // Same value as Java client
+ASSERT_EQ(2110152746, hash.makeHash(k1));
+ASSERT_EQ(147994, hash.makeHash(k2));
 
 Review comment:
   > We need to test if the hashes generated from keys whose length is not a 
multiple of 4 are correct.
   
   I got it, thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new issue #1797: Proxy doesn't strip the request URL for admin requests correctly

[GitBox]

ivankelly opened a new issue #1797: Proxy doesn't strip the request URL for 
admin requests correctly
URL: https://github.com/apache/incubator-pulsar/issues/1797
 
 
    Expected behavior
   
   Using proxy to do admin tasks should work.
   
    Actual behavior
   
   On client
   ```
   org.apache.pulsar.client.admin.PulsarAdminException$NotAllowedException: 
HTTP 405 HTTP method PUT is not supported by this URL
   ```
   
   From proxy log.
   ```
   09:33:50.627 [pulsar-external-web-4-8] INFO  
org.eclipse.jetty.server.RequestLog - 172.29.0.1 - - [17/May/2018:09:33:50 
+] "PUT 
//172.29.0.4:8080/admin/v2/persistent/s3-offload-test/ns1/topic1/offload 
HTTP/1.1" 405 406 "-" "Jersey/2.25 (HttpUrlConnection 1.8.0_151)" 10
   ```
   
    Steps to reproduce
   
   Set up a broker with mostly default config (just set up zk stuff)
   Set up a proxy with mostly default config (just set zk and brokerServiceURL)
   
   I had 
   ```
   brokerServiceURL=http://pulsar-broker1:8080
   ```
   where pulsar-broker1 resolves to 172.29.0.5
   172.29.0.4 is the ip of the proxy itself.
   
   Admin client is created with:
   ```
   PulsarAdmin admin = new PulsarAdmin(new URL("http://172.29.0.4:8080;), "", 
"")
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] massakam commented on a change in pull request #1795: Fix C++ MurmurHash3 algorithm

[GitBox]

massakam commented on a change in pull request #1795: Fix C++ MurmurHash3 
algorithm
URL: https://github.com/apache/incubator-pulsar/pull/1795#discussion_r188892224
 
 

 ##
 File path: pulsar-client-cpp/tests/HashTest.cc
 ##
 @@ -58,10 +58,18 @@ TEST(HashTest, testJavaStringHash) {
 
 TEST(HashTest, testMurmur3_32Hash) {
 Murmur3_32Hash hash;
+std::string k1 = "k1";
+std::string k2 = "k2";
 std::string key1 = "key1";
 std::string key2 = "key2";
+std::string key01 = "key01";
+std::string key02 = "key02";
 
 // Same value as Java client
+ASSERT_EQ(2110152746, hash.makeHash(k1));
+ASSERT_EQ(147994, hash.makeHash(k2));
 
 Review comment:
   Added the same test cases to Java.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new issue #1796: Proxy cannot be configured to work as proxy for both admin and client requests

[GitBox]

ivankelly opened a new issue #1796: Proxy cannot be configured to work as proxy 
for both admin and client requests
URL: https://github.com/apache/incubator-pulsar/issues/1796
 
 
    Expected behavior
   
   When I start a proxy, I should be able to point both a client and an admin 
client at it, and expect it to work.
   
    Actual behavior
   
   By default, admin client will not work, as brokerServiceURL isn't configured 
(this should come from zookeeper, the key isn't even in the default config 
file). 
   
   If I configure brokerServiceURL, then admin client works, but the client 
itself doesn't work, as it's trying to send binary protocol requests to a http 
endpoint.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed pull request #1784: PIP-17: impl deleteOffloaded() for S3ManagedLedgerOffloader

[GitBox]

sijie closed pull request #1784: PIP-17: impl deleteOffloaded() for 
S3ManagedLedgerOffloader
URL: https://github.com/apache/incubator-pulsar/pull/1784
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
index 4d5b388a59..7a73a3bbc8 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
@@ -18,13 +18,12 @@
  */
 package org.apache.pulsar.broker.s3offload;
 
-import com.amazonaws.AmazonServiceException;
-import com.amazonaws.SdkClientException;
 import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
 import com.amazonaws.services.s3.AmazonS3;
 import com.amazonaws.services.s3.AmazonS3ClientBuilder;
 import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
 import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.DeleteObjectsRequest;
 import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
 import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
 import com.amazonaws.services.s3.model.ObjectMetadata;
@@ -212,7 +211,19 @@ static String indexBlockOffloadKey(long ledgerId, UUID 
uuid) {
 @Override
 public CompletableFuture deleteOffloaded(long ledgerId, UUID uid) {
 CompletableFuture promise = new CompletableFuture<>();
-promise.completeExceptionally(new UnsupportedOperationException());
+scheduler.submit(() -> {
+try {
+
+s3client.deleteObjects(new DeleteObjectsRequest(bucket)
+.withKeys(dataBlockOffloadKey(ledgerId, uid), 
indexBlockOffloadKey(ledgerId, uid)));
+promise.complete(null);
+} catch (Throwable t) {
+log.error("Failed delete s3 Object ", t);
+promise.completeExceptionally(t);
+return;
+}
+});
+
 return promise;
 }
 }
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloaderTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloaderTest.java
index f9a043b179..ed44ddc586 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloaderTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloaderTest.java
@@ -25,12 +25,9 @@
 import com.amazonaws.AmazonServiceException;
 import com.amazonaws.services.s3.AmazonS3;
 import io.netty.util.concurrent.DefaultThreadFactory;
-
-import java.io.DataInputStream;
 import java.lang.reflect.Method;
 import java.util.HashMap;
 import java.util.Iterator;
-import java.util.Map;
 import java.util.Random;
 import java.util.UUID;
 import java.util.concurrent.ExecutionException;
@@ -38,7 +35,6 @@
 import java.util.concurrent.ScheduledExecutorService;
 
 import lombok.extern.slf4j.Slf4j;
-
 import org.apache.bookkeeper.client.BKException;
 import org.apache.bookkeeper.client.BookKeeper;
 import org.apache.bookkeeper.client.LedgerHandle;
@@ -52,9 +48,7 @@
 import org.apache.pulsar.broker.PulsarServerException;
 import org.apache.pulsar.broker.ServiceConfiguration;
 import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
-import 
org.apache.pulsar.broker.s3offload.impl.BlockAwareSegmentInputStreamImpl;
 import org.apache.pulsar.broker.s3offload.impl.DataBlockHeaderImpl;
-import org.apache.pulsar.broker.s3offload.impl.OffloadIndexBlockImpl;
 import org.mockito.Mockito;
 import org.testng.Assert;
 import org.testng.annotations.Test;
@@ -370,5 +364,54 @@ public void testOffloadReadInvalidEntryIds() throws 
Exception {
 } catch (BKException.BKIncorrectParameterException e) {
 }
 }
+
+@Test
+public void testDeleteOffloaded() throws Exception {
+int maxBlockSize = 1024;
+int entryCount = 3;
+ReadHandle readHandle = buildReadHandle(maxBlockSize, entryCount);
+UUID uuid = UUID.randomUUID();
+LedgerOffloader offloader = new S3ManagedLedgerOffloader(s3client, 
BUCKET, scheduler, maxBlockSize, DEFAULT_READ_BUFFER_SIZE);
+
+// verify object exist after offload
+offloader.offload(readHandle, uuid, new HashMap<>()).get();
+Assert.assertTrue(s3client.doesObjectExist(BUCKET, 
dataBlockOffloadKey(readHandle.getId(), uuid)));
+Assert.assertTrue(s3client.doesObjectExist(BUCKET, 
indexBlockOffloadKey(readHandle.getId(), uuid)));
+
+// verify object 

[incubator-pulsar] branch asf-site updated: Updated site at revision 971f76b

[mmerli]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 984bf75  Updated site at revision 971f76b
984bf75 is described below

commit 984bf754184c47e4e020f13d5c8e9ec1a88b7421
Author: jenkins 
AuthorDate: Thu May 17 08:34:48 2018 +

Updated site at revision 971f76b
---
 content/api/cpp/authentication_8h_source.html  |  2 +-
 content/api/cpp/client_8h_source.html  |  2 +-
 .../api/cpp/client__configuration_8h_source.html   |  2 +-
 content/api/cpp/consumer_8h_source.html|  2 +-
 .../api/cpp/consumer__configuration_8h_source.html |  2 +-
 content/api/cpp/message_8h_source.html |  2 +-
 content/api/cpp/message__id_8h_source.html |  2 +-
 content/api/cpp/message__router_8h_source.html |  2 +-
 content/api/cpp/producer_8h_source.html|  2 +-
 .../api/cpp/producer__configuration_8h_source.html |  2 +-
 content/api/cpp/reader_8h_source.html  |  2 +-
 .../api/cpp/reader__configuration_8h_source.html   |  2 +-
 content/api/cpp/result_8h_source.html  |  2 +-
 .../docs/latest/adaptors/KafkaWrapper/index.html   |  2 +-
 .../docs/latest/adaptors/PulsarSpark/index.html| 10 -
 .../docs/latest/adaptors/PulsarStorm/index.html|  8 
 content/docs/latest/admin-api/brokers/index.html   |  2 +-
 content/docs/latest/admin-api/clusters/index.html  |  2 +-
 .../docs/latest/admin-api/namespaces/index.html|  2 +-
 .../admin-api/non-persistent-topics/index.html |  2 +-
 content/docs/latest/admin-api/overview/index.html  | 12 +--
 .../latest/admin-api/partitioned-topics/index.html |  2 +-
 .../docs/latest/admin-api/permissions/index.html   |  2 +-
 .../latest/admin-api/persistent-topics/index.html  |  2 +-
 content/docs/latest/admin-api/tenants/index.html   |  2 +-
 content/docs/latest/admin/Authz/index.html | 22 ++--
 content/docs/latest/admin/Dashboard/index.html |  2 +-
 .../docs/latest/admin/GeoReplication/index.html|  2 +-
 .../docs/latest/admin/LoadDistribution/index.html  |  2 +-
 content/docs/latest/admin/Proxy/index.html |  2 +-
 content/docs/latest/admin/Stats/index.html |  2 +-
 .../latest/admin/ZooKeeperBookKeeper/index.html|  2 +-
 content/docs/latest/clients/Cpp/index.html | 10 -
 content/docs/latest/clients/Java/index.html| 14 ++---
 content/docs/latest/clients/Python/index.html  | 10 -
 content/docs/latest/clients/WebSocket/index.html   | 10 -
 .../docs/latest/cookbooks/Encryption/index.html|  8 
 .../latest/cookbooks/PartitionedTopics/index.html  | 16 +++
 .../latest/cookbooks/RetentionExpiry/index.html| 16 +++
 .../cookbooks/message-deduplication/index.html | 12 +--
 .../docs/latest/cookbooks/message-queue/index.html | 16 +++
 .../cookbooks/non-persistent-messaging/index.html  |  2 +-
 .../docs/latest/deployment/Kubernetes/index.html   |  6 +++---
 .../docs/latest/deployment/Monitoring/index.html   |  2 +-
 .../docs/latest/deployment/aws-cluster/index.html  |  6 +++---
 content/docs/latest/deployment/cluster/index.html  |  6 +++---
 content/docs/latest/deployment/dcos/index.html |  2 +-
 content/docs/latest/deployment/instance/index.html |  6 +++---
 content/docs/latest/functions/api/index.html   |  2 +-
 .../docs/latest/functions/deployment/index.html|  2 +-
 .../docs/latest/functions/guarantees/index.html|  2 +-
 content/docs/latest/functions/metrics/index.html   |  2 +-
 content/docs/latest/functions/overview/index.html  |  2 +-
 .../docs/latest/functions/quickstart/index.html|  2 +-
 .../docs/latest/getting-started/Clients/index.html |  2 +-
 .../ConceptsAndArchitecture/index.html |  6 +++---
 .../latest/getting-started/LocalCluster/index.html |  6 +++---
 .../latest/getting-started/Pulsar-2.0/index.html   |  4 ++--
 .../docs/latest/getting-started/docker/index.html  |  6 +++---
 .../docs/latest/project/BinaryProtocol/index.html  |  6 +++---
 content/docs/latest/project/Codebase/index.html|  2 +-
 .../latest/project/ModularLoadManager/index.html   |  2 +-
 .../docs/latest/project/SimulationTools/index.html |  4 ++--
 .../docs/latest/project/schema-storage/index.html  |  6 +++---
 content/docs/latest/reference/CliTools/index.html  | 18 
 .../docs/latest/reference/Configuration/index.html |  2 +-
 content/docs/latest/reference/RestApi/index.html   |  2 +-
 content/ja/adaptors/PulsarSpark/index.html |  8 
 content/ja/adaptors/PulsarStorm/index.html |  6 +++---
 content/ja/admin/AdminInterface/index.html | 14 ++---
 content/ja/admin/Authz/index.html  | 24 +++---
 

[incubator-pulsar] branch master updated: PIP-17: impl deleteOffloaded() for S3ManagedLedgerOffloader (#1784)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new a6544bd  PIP-17: impl deleteOffloaded() for S3ManagedLedgerOffloader 
(#1784)
a6544bd is described below

commit a6544bdb82de3056c6350ae9ad389a0f0e5fd2f6
Author: Jia Zhai 
AuthorDate: Thu May 17 16:35:13 2018 +0800

PIP-17: impl deleteOffloaded() for S3ManagedLedgerOffloader (#1784)

* s3 delete

* change following @ivan's comments

* fix issue after merge master
---
 .../broker/s3offload/S3ManagedLedgerOffloader.java | 17 +--
 .../s3offload/S3ManagedLedgerOffloaderTest.java| 55 +++---
 2 files changed, 63 insertions(+), 9 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
index 4d5b388..7a73a3b 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
@@ -18,13 +18,12 @@
  */
 package org.apache.pulsar.broker.s3offload;
 
-import com.amazonaws.AmazonServiceException;
-import com.amazonaws.SdkClientException;
 import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
 import com.amazonaws.services.s3.AmazonS3;
 import com.amazonaws.services.s3.AmazonS3ClientBuilder;
 import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
 import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.DeleteObjectsRequest;
 import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
 import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
 import com.amazonaws.services.s3.model.ObjectMetadata;
@@ -212,7 +211,19 @@ public class S3ManagedLedgerOffloader implements 
LedgerOffloader {
 @Override
 public CompletableFuture deleteOffloaded(long ledgerId, UUID uid) {
 CompletableFuture promise = new CompletableFuture<>();
-promise.completeExceptionally(new UnsupportedOperationException());
+scheduler.submit(() -> {
+try {
+
+s3client.deleteObjects(new DeleteObjectsRequest(bucket)
+.withKeys(dataBlockOffloadKey(ledgerId, uid), 
indexBlockOffloadKey(ledgerId, uid)));
+promise.complete(null);
+} catch (Throwable t) {
+log.error("Failed delete s3 Object ", t);
+promise.completeExceptionally(t);
+return;
+}
+});
+
 return promise;
 }
 }
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloaderTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloaderTest.java
index f9a043b..ed44ddc 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloaderTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloaderTest.java
@@ -25,12 +25,9 @@ import static org.mockito.Matchers.any;
 import com.amazonaws.AmazonServiceException;
 import com.amazonaws.services.s3.AmazonS3;
 import io.netty.util.concurrent.DefaultThreadFactory;
-
-import java.io.DataInputStream;
 import java.lang.reflect.Method;
 import java.util.HashMap;
 import java.util.Iterator;
-import java.util.Map;
 import java.util.Random;
 import java.util.UUID;
 import java.util.concurrent.ExecutionException;
@@ -38,7 +35,6 @@ import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 
 import lombok.extern.slf4j.Slf4j;
-
 import org.apache.bookkeeper.client.BKException;
 import org.apache.bookkeeper.client.BookKeeper;
 import org.apache.bookkeeper.client.LedgerHandle;
@@ -52,9 +48,7 @@ import org.apache.bookkeeper.mledger.LedgerOffloader;
 import org.apache.pulsar.broker.PulsarServerException;
 import org.apache.pulsar.broker.ServiceConfiguration;
 import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
-import 
org.apache.pulsar.broker.s3offload.impl.BlockAwareSegmentInputStreamImpl;
 import org.apache.pulsar.broker.s3offload.impl.DataBlockHeaderImpl;
-import org.apache.pulsar.broker.s3offload.impl.OffloadIndexBlockImpl;
 import org.mockito.Mockito;
 import org.testng.Assert;
 import org.testng.annotations.Test;
@@ -370,5 +364,54 @@ class S3ManagedLedgerOffloaderTest extends S3TestBase {
 } catch (BKException.BKIncorrectParameterException e) {
 }
 }
+
+@Test
+public void testDeleteOffloaded() throws Exception {
+int maxBlockSize = 1024;
+int entryCount = 3;
+ReadHandle readHandle = 

[GitHub] zhaijack commented on issue #1784: PIP-17: impl deleteOffloaded() for S3ManagedLedgerOffloader

[GitBox]

zhaijack commented on issue #1784: PIP-17: impl deleteOffloaded() for 
S3ManagedLedgerOffloader
URL: https://github.com/apache/incubator-pulsar/pull/1784#issuecomment-389781122
 
 
   @sijie Would you please help review this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] massakam commented on a change in pull request #1795: Fix C++ MurmurHash3 algorithm

[GitBox]

massakam commented on a change in pull request #1795: Fix C++ MurmurHash3 
algorithm
URL: https://github.com/apache/incubator-pulsar/pull/1795#discussion_r188864565
 
 

 ##
 File path: pulsar-client-cpp/tests/HashTest.cc
 ##
 @@ -58,10 +58,18 @@ TEST(HashTest, testJavaStringHash) {
 
 TEST(HashTest, testMurmur3_32Hash) {
 Murmur3_32Hash hash;
+std::string k1 = "k1";
+std::string k2 = "k2";
 std::string key1 = "key1";
 std::string key2 = "key2";
+std::string key01 = "key01";
+std::string key02 = "key02";
 
 // Same value as Java client
+ASSERT_EQ(2110152746, hash.makeHash(k1));
+ASSERT_EQ(147994, hash.makeHash(k2));
 
 Review comment:
   That's because this problem occurs when the key length is not a multiple of 
4.
   We need to test if the hashes generated from keys whose length is not a 
multiple of 4 are correct.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nkurihar commented on a change in pull request #1795: Fix C++ MurmurHash3 algorithm

[GitBox]

nkurihar commented on a change in pull request #1795: Fix C++ MurmurHash3 
algorithm
URL: https://github.com/apache/incubator-pulsar/pull/1795#discussion_r188861733
 
 

 ##
 File path: pulsar-client-cpp/tests/HashTest.cc
 ##
 @@ -58,10 +58,18 @@ TEST(HashTest, testJavaStringHash) {
 
 TEST(HashTest, testMurmur3_32Hash) {
 Murmur3_32Hash hash;
+std::string k1 = "k1";
+std::string k2 = "k2";
 std::string key1 = "key1";
 std::string key2 = "key2";
+std::string key01 = "key01";
+std::string key02 = "key02";
 
 // Same value as Java client
+ASSERT_EQ(2110152746, hash.makeHash(k1));
+ASSERT_EQ(147994, hash.makeHash(k2));
 
 Review comment:
   What is the purpose of adding these test cases(k1, k2, key01, key02)?
   If you want to compare more values with Java client, I think you should add 
the same test cases to 
[HashTest.java](https://github.com/apache/incubator-pulsar/blob/8d159efa5a4f25191fade5a9fddb45a245ef2e6b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java#L41-L48).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nkurihar commented on a change in pull request #1795: Fix C++ MurmurHash3 algorithm

[GitBox]

nkurihar commented on a change in pull request #1795: Fix C++ MurmurHash3 
algorithm
URL: https://github.com/apache/incubator-pulsar/pull/1795#discussion_r188861733
 
 

 ##
 File path: pulsar-client-cpp/tests/HashTest.cc
 ##
 @@ -58,10 +58,18 @@ TEST(HashTest, testJavaStringHash) {
 
 TEST(HashTest, testMurmur3_32Hash) {
 Murmur3_32Hash hash;
+std::string k1 = "k1";
+std::string k2 = "k2";
 std::string key1 = "key1";
 std::string key2 = "key2";
+std::string key01 = "key01";
+std::string key02 = "key02";
 
 // Same value as Java client
+ASSERT_EQ(2110152746, hash.makeHash(k1));
+ASSERT_EQ(147994, hash.makeHash(k2));
 
 Review comment:
   What is the purpose of adding these test cases(k1, k2, key01, key02)?
   If you want to compare more values with Java client, I think you should add 
the same test cases to 
[HashTest.java](https://github.com/apache/incubator-pulsar/blob/8d159efa5a4f25191fade5a9fddb45a245ef2e6b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/HashTest.java#L41-L48)
 ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services