[GitHub] merlimat closed pull request #2572: Shorten the timeout value of C++ ZTS client
merlimat closed pull request #2572: Shorten the timeout value of C++ ZTS client
URL: https://github.com/apache/incubator-pulsar/pull/2572
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/pulsar-client-cpp/lib/auth/athenz/ZTSClient.cc
b/pulsar-client-cpp/lib/auth/athenz/ZTSClient.cc
index 93010affb8..3097c30c65 100644
--- a/pulsar-client-cpp/lib/auth/athenz/ZTSClient.cc
+++ b/pulsar-client-cpp/lib/auth/athenz/ZTSClient.cc
@@ -45,7 +45,7 @@ namespace pulsar {
const static std::string DEFAULT_PRINCIPAL_HEADER = "Athenz-Principal-Auth";
const static std::string DEFAULT_ROLE_HEADER = "Athenz-Role-Auth";
-const static int REQUEST_TIMEOUT = 1;
+const static int REQUEST_TIMEOUT = 3;
const static int DEFAULT_TOKEN_EXPIRATION_TIME_SEC = 3600;
const static int MIN_TOKEN_EXPIRATION_TIME_SEC = 900;
const static int MAX_HTTP_REDIRECTS = 20;
@@ -272,7 +272,7 @@ const std::string ZTSClient::getRoleToken() const {
curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1L);
// Timer
-curl_easy_setopt(handle, CURLOPT_TIMEOUT, REQUEST_TIMEOUT);
+curl_easy_setopt(handle, CURLOPT_TIMEOUT_MS, REQUEST_TIMEOUT);
// Redirects
curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1L);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-pulsar] branch master updated: Shorten the timeout value of C++ ZTS client (#2572)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new f076974 Shorten the timeout value of C++ ZTS client (#2572)
f076974 is described below
commit f076974935feeece00d32094c16572ac4f45244c
Author: massakam
AuthorDate: Thu Sep 20 09:11:45 2018 +0900
Shorten the timeout value of C++ ZTS client (#2572)
* Shorten the timeout value of C++ ZTS client
* Set REQUEST_TIMEOUT to 3ms
---
pulsar-client-cpp/lib/auth/athenz/ZTSClient.cc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pulsar-client-cpp/lib/auth/athenz/ZTSClient.cc
b/pulsar-client-cpp/lib/auth/athenz/ZTSClient.cc
index 93010af..3097c30 100644
--- a/pulsar-client-cpp/lib/auth/athenz/ZTSClient.cc
+++ b/pulsar-client-cpp/lib/auth/athenz/ZTSClient.cc
@@ -45,7 +45,7 @@ namespace pulsar {
const static std::string DEFAULT_PRINCIPAL_HEADER = "Athenz-Principal-Auth";
const static std::string DEFAULT_ROLE_HEADER = "Athenz-Role-Auth";
-const static int REQUEST_TIMEOUT = 1;
+const static int REQUEST_TIMEOUT = 3;
const static int DEFAULT_TOKEN_EXPIRATION_TIME_SEC = 3600;
const static int MIN_TOKEN_EXPIRATION_TIME_SEC = 900;
const static int MAX_HTTP_REDIRECTS = 20;
@@ -272,7 +272,7 @@ const std::string ZTSClient::getRoleToken() const {
curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1L);
// Timer
-curl_easy_setopt(handle, CURLOPT_TIMEOUT, REQUEST_TIMEOUT);
+curl_easy_setopt(handle, CURLOPT_TIMEOUT_MS, REQUEST_TIMEOUT);
// Redirects
curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1L);
[GitHub] david-streamlio opened a new pull request #2615: Azure offloader
david-streamlio opened a new pull request #2615: Azure offloader URL: https://github.com/apache/incubator-pulsar/pull/2615 ### Motivation Added Azure offloader for Pulsar Tiered Storage ### Modifications Refactored the tiered storage configuration tree to make it easier to add other JCloud Storage Providers in the future ### Result Users will be able to use their Azure Storage accounts for tiered storage of Pulsar data. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jiazhai opened a new pull request #2614: Debezium: add PulsarDatabaseHistory for debezium
jiazhai opened a new pull request #2614: Debezium: add PulsarDatabaseHistory for debezium URL: https://github.com/apache/incubator-pulsar/pull/2614 ### Motivation add PulsarDatabaseHistory for debezium ### Modifications add PulsarDatabaseHistory for debezium and test for it. ### Result ut pass This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Enforce boost-python was found by CMake (#2608)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new d536d66 Enforce boost-python was found by CMake (#2608)
d536d66 is described below
commit d536d6669eb79c7b4235403c43ef5d6cf3b79dcd
Author: Matteo Merli
AuthorDate: Wed Sep 19 17:03:52 2018 -0700
Enforce boost-python was found by CMake (#2608)
* Enforce boost-python was found by CMake
* Fix for linux build as well
---
pulsar-client-cpp/docker/build-wheels.sh | 1 +
pulsar-client-cpp/python/CMakeLists.txt | 9 +
2 files changed, 10 insertions(+)
diff --git a/pulsar-client-cpp/docker/build-wheels.sh
b/pulsar-client-cpp/docker/build-wheels.sh
index ebbb441..3292519 100755
--- a/pulsar-client-cpp/docker/build-wheels.sh
+++ b/pulsar-client-cpp/docker/build-wheels.sh
@@ -33,6 +33,7 @@ PYTHON_VERSIONS=(
'3.4 cp34-cp34m'
'3.5 cp35-cp35m'
'3.6 cp36-cp36m'
+ '3.7 cp37-cp37m'
)
function contains() {
diff --git a/pulsar-client-cpp/python/CMakeLists.txt
b/pulsar-client-cpp/python/CMakeLists.txt
index 9ddbc01..2c51f6d 100644
--- a/pulsar-client-cpp/python/CMakeLists.txt
+++ b/pulsar-client-cpp/python/CMakeLists.txt
@@ -48,8 +48,17 @@ endif()
set(PYTHON_WRAPPER_LIBS ${Boost_PYTHON_LIBRARY} ${Boost_PYTHON3_LIBRARY}
${Boost_PYTHON27-MT_LIBRARY}
${Boost_PYTHON37-MT_LIBRARY})
+if (APPLE)
+set(PYTHON_WRAPPER_LIBS ${PYTHON_WRAPPER_LIBS}
+ ${Boost_PYTHON27-MT_LIBRARY_RELEASE}
${Boost_PYTHON37-MT_LIBRARY_RELEASE})
+endif()
+
message(STATUS "Using Boost Python libs: ${PYTHON_WRAPPER_LIBS}")
+if (NOT PYTHON_WRAPPER_LIBS)
+MESSAGE(FATAL_ERROR "Could not find Boost Python library")
+endif ()
+
if (APPLE)
target_link_libraries(_pulsar -Wl,-all_load pulsarStatic
${PYTHON_WRAPPER_LIBS})
else ()
[GitHub] merlimat closed pull request #2608: Enforce boost-python was found by CMake
merlimat closed pull request #2608: Enforce boost-python was found by CMake
URL: https://github.com/apache/incubator-pulsar/pull/2608
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/pulsar-client-cpp/docker/build-wheels.sh
b/pulsar-client-cpp/docker/build-wheels.sh
index ebbb441cac..32925194ec 100755
--- a/pulsar-client-cpp/docker/build-wheels.sh
+++ b/pulsar-client-cpp/docker/build-wheels.sh
@@ -33,6 +33,7 @@ PYTHON_VERSIONS=(
'3.4 cp34-cp34m'
'3.5 cp35-cp35m'
'3.6 cp36-cp36m'
+ '3.7 cp37-cp37m'
)
function contains() {
diff --git a/pulsar-client-cpp/python/CMakeLists.txt
b/pulsar-client-cpp/python/CMakeLists.txt
index 9ddbc017c1..2c51f6d0b0 100644
--- a/pulsar-client-cpp/python/CMakeLists.txt
+++ b/pulsar-client-cpp/python/CMakeLists.txt
@@ -48,8 +48,17 @@ endif()
set(PYTHON_WRAPPER_LIBS ${Boost_PYTHON_LIBRARY} ${Boost_PYTHON3_LIBRARY}
${Boost_PYTHON27-MT_LIBRARY}
${Boost_PYTHON37-MT_LIBRARY})
+if (APPLE)
+set(PYTHON_WRAPPER_LIBS ${PYTHON_WRAPPER_LIBS}
+ ${Boost_PYTHON27-MT_LIBRARY_RELEASE}
${Boost_PYTHON37-MT_LIBRARY_RELEASE})
+endif()
+
message(STATUS "Using Boost Python libs: ${PYTHON_WRAPPER_LIBS}")
+if (NOT PYTHON_WRAPPER_LIBS)
+MESSAGE(FATAL_ERROR "Could not find Boost Python library")
+endif ()
+
if (APPLE)
target_link_libraries(_pulsar -Wl,-all_load pulsarStatic
${PYTHON_WRAPPER_LIBS})
else ()
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] rdhabalia opened a new pull request #2613: Avoid scheduling heartbeat function if owner-worker not available
rdhabalia opened a new pull request #2613: Avoid scheduling heartbeat function if owner-worker not available URL: https://github.com/apache/incubator-pulsar/pull/2613 ### Motivation Handle heartbeat function if owner-worker is not available. ### Modifications Function scheduling handles heartbeat function if owner-worker is not available. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jiazhai commented on issue #2590: Issue #2584: unacked message is not redelivered on time
jiazhai commented on issue #2590: Issue #2584: unacked message is not redelivered on time URL: https://github.com/apache/incubator-pulsar/pull/2590#issuecomment-422994297 seems some ut need change, will look into it later. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jiazhai removed a comment on issue #2606: Add support for dead letter topics for java functions
jiazhai removed a comment on issue #2606: Add support for dead letter topics for java functions URL: https://github.com/apache/incubator-pulsar/pull/2606#issuecomment-422994155 Seems some ut need change, will look into it later. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jiazhai commented on issue #2606: Add support for dead letter topics for java functions
jiazhai commented on issue #2606: Add support for dead letter topics for java functions URL: https://github.com/apache/incubator-pulsar/pull/2606#issuecomment-422994155 Seems some ut need change, will look into it later. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2605: implement topic routing on a per record basis
jerrypeng commented on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422990454 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on a change in pull request #2593: Add support for running python functions with wheel file
jerrypeng commented on a change in pull request #2593: Add support for running
python functions with wheel file
URL: https://github.com/apache/incubator-pulsar/pull/2593#discussion_r218982622
##
File path:
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
##
@@ -229,6 +229,11 @@ void processArguments() throws Exception {
description = "Path to the main Python file for the function
(if the function is written in Python)",
listConverter = StringConverter.class)
protected String pyFile;
+@Parameter(
+names = "--pywheel",
+description = "Path to the Python wheel file for the function
(if the function is submitted as Python Wheel)",
+listConverter = StringConverter.class)
Review comment:
why do we need a listConverter?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sijie closed issue #2007: Implement an easier way to get the topic information from consumed message
sijie closed issue #2007: Implement an easier way to get the topic information from consumed message URL: https://github.com/apache/incubator-pulsar/issues/2007 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2007: Implement an easier way to get the topic information from consumed message
sijie commented on issue #2007: Implement an easier way to get the topic information from consumed message URL: https://github.com/apache/incubator-pulsar/issues/2007#issuecomment-422970257 This has been implemented and merged. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2009: Unable to get messages from non-persistent topics
sijie commented on issue #2009: Unable to get messages from non-persistent topics URL: https://github.com/apache/incubator-pulsar/issues/2009#issuecomment-422970028 This is being fixed by #2025, will be included in the upcoming release. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #2576: Pulsar client consume command fails with java.lang.IllegalArgumentException: port out of range:-1
dsambandam commented on issue #2576: Pulsar client consume command fails with java.lang.IllegalArgumentException: port out of range:-1 URL: https://github.com/apache/incubator-pulsar/issues/2576#issuecomment-422969989 @sijie I don't have any special character in my service urls. In that case my command that runs fine with no authentication enabled should also fail right ? That gets completed successfully with no issues. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2013: Unable to consume messages from a partition
sijie commented on issue #2013: Unable to consume messages from a partition URL: https://github.com/apache/incubator-pulsar/issues/2013#issuecomment-422969773 I think this is a duplicate of #2431 , closing this one. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed issue #2013: Unable to consume messages from a partition
sijie closed issue #2013: Unable to consume messages from a partition URL: https://github.com/apache/incubator-pulsar/issues/2013 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2033: Python pulsar-client lacks multi-topic support
sijie commented on issue #2033: Python pulsar-client lacks multi-topic support URL: https://github.com/apache/incubator-pulsar/issues/2033#issuecomment-422969364 multi-topic and regex subscription support is added to python. it is actually released in 2.1.1 and will also be in 2.2.0 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed issue #2033: Python pulsar-client lacks multi-topic support
sijie closed issue #2033: Python pulsar-client lacks multi-topic support URL: https://github.com/apache/incubator-pulsar/issues/2033 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener
sijie commented on issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener URL: https://github.com/apache/incubator-pulsar/issues/2574#issuecomment-422968685 This is fixed by #2583 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2576: Pulsar client consume command fails with java.lang.IllegalArgumentException: port out of range:-1
sijie commented on issue #2576: Pulsar client consume command fails with java.lang.IllegalArgumentException: port out of range:-1 URL: https://github.com/apache/incubator-pulsar/issues/2576#issuecomment-422968544 @dsambandam I think the exception is quite similar ast the issue reported here: #1022 can you inspect your tls related service urls and make sure they don't have any special characters at the end of the hostname? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener
sijie closed issue #2574: Timeout message not get redeliver in TopicsConsumer when use message listener URL: https://github.com/apache/incubator-pulsar/issues/2574 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1016: Support bytes message key
sijie commented on issue #1016: Support bytes message key URL: https://github.com/apache/incubator-pulsar/issues/1016#issuecomment-422966131 this is being implemented in #2612 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2593: Add support for running python functions with wheel file
srkukarni commented on issue #2593: Add support for running python functions with wheel file URL: https://github.com/apache/incubator-pulsar/pull/2593#issuecomment-422965788 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2504: Add Presto Sql Test
aahmed-se commented on issue #2504: Add Presto Sql Test URL: https://github.com/apache/incubator-pulsar/pull/2504#issuecomment-422963308 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2578: Add support for schema extraction from a jar
aahmed-se commented on issue #2578: Add support for schema extraction from a jar URL: https://github.com/apache/incubator-pulsar/pull/2578#issuecomment-422963330 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2611: Make dockerUtils use container name for test exec
aahmed-se commented on issue #2611: Make dockerUtils use container name for test exec URL: https://github.com/apache/incubator-pulsar/pull/2611#issuecomment-422962898 the log configure does not, the DockerUtils changes fixes that issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2611: Make dockerUtils use container name for test exec
sijie commented on issue #2611: Make dockerUtils use container name for test exec URL: https://github.com/apache/incubator-pulsar/pull/2611#issuecomment-422954395 > we are reducing the number of log line elements since we are not looking at multithreaded server components but single threaded test instances well, I think it is debatable. more information is generally helping with debugging things. is there any particular reasons that you have to reduces the log line elements? > also we are making the log files easier to follow bu using the containername how does the log configure help that? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #2612: Allow byte[] keys for messages (#1016)
sijie commented on a change in pull request #2612: Allow byte[] keys for messages (#1016) URL: https://github.com/apache/incubator-pulsar/pull/2612#discussion_r218956364 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/TypedMessageBuilder.java ## @@ -89,6 +89,14 @@ */ TypedMessageBuilder key(String key); +/** + * Sets the bytes of the key of the message for routing policy. + * Internally the bytes will be base64 encoded. + * + * @param key routing key for message, in byte array form + */ +TypedMessageBuilder keyBytes(byte[] key); Review comment: any particular reason on naming this `keyBytes` instead of `key`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
svn commit: r29520 - in /release/incubator/pulsar: pulsar-2.1.0-incubating/ pulsar-2.1.1-incubating/ pulsar-2.1.1-incubating/DEB/ pulsar-2.1.1-incubating/RPMS/ pulsar-2.1.1-incubating/RPMS/repodata/
Author: jerrypeng Date: Wed Sep 19 20:19:08 2018 New Revision: 29520 Log: Release v2.1.1-incubating Added: release/incubator/pulsar/pulsar-2.1.1-incubating/ release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/ release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/Packages.gz (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/Packages.gz.asc release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/Packages.gz.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/apache-pulsar-client-dev.deb (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/apache-pulsar-client-dev.deb.asc release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/apache-pulsar-client-dev.deb.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/apache-pulsar-client.deb (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/apache-pulsar-client.deb.asc release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/apache-pulsar-client.deb.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/go1.9.1.linux-amd64.tar.gz (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/go1.9.1.linux-amd64.tar.gz.asc release/incubator/pulsar/pulsar-2.1.1-incubating/DEB/go1.9.1.linux-amd64.tar.gz.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/ release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/apache-pulsar-client-2.1.1-1_incubating.x86_64.rpm (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/apache-pulsar-client-2.1.1-1_incubating.x86_64.rpm.asc release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/apache-pulsar-client-2.1.1-1_incubating.x86_64.rpm.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/apache-pulsar-client-debuginfo-2.1.1-1_incubating.x86_64.rpm (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/apache-pulsar-client-debuginfo-2.1.1-1_incubating.x86_64.rpm.asc release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/apache-pulsar-client-debuginfo-2.1.1-1_incubating.x86_64.rpm.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/apache-pulsar-client-devel-2.1.1-1_incubating.x86_64.rpm (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/apache-pulsar-client-devel-2.1.1-1_incubating.x86_64.rpm.asc release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/apache-pulsar-client-devel-2.1.1-1_incubating.x86_64.rpm.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/ release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/1f13f20740e4991769c6777c41e826ec04607c1a53a24b45d9ffc3a988a1a42c-filelists.xml.gz (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/1f13f20740e4991769c6777c41e826ec04607c1a53a24b45d9ffc3a988a1a42c-filelists.xml.gz.asc release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/1f13f20740e4991769c6777c41e826ec04607c1a53a24b45d9ffc3a988a1a42c-filelists.xml.gz.sha1 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/1f13f20740e4991769c6777c41e826ec04607c1a53a24b45d9ffc3a988a1a42c-filelists.xml.gz.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/29594b740e432893aca1a91c6cbd1572a3cf4333862c6a98ff731a189d1cdb27-other.xml.gz (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/29594b740e432893aca1a91c6cbd1572a3cf4333862c6a98ff731a189d1cdb27-other.xml.gz.asc release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/29594b740e432893aca1a91c6cbd1572a3cf4333862c6a98ff731a189d1cdb27-other.xml.gz.sha1 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/29594b740e432893aca1a91c6cbd1572a3cf4333862c6a98ff731a189d1cdb27-other.xml.gz.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/8e286e31910c9f58295862fdc6617993a1dd83fa5e1964df9e08664d696aaf2d-other.sqlite.bz2 (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/8e286e31910c9f58295862fdc6617993a1dd83fa5e1964df9e08664d696aaf2d-other.sqlite.bz2.asc release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/8e286e31910c9f58295862fdc6617993a1dd83fa5e1964df9e08664d696aaf2d-other.sqlite.bz2.sha1 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/8e286e31910c9f58295862fdc6617993a1dd83fa5e1964df9e08664d696aaf2d-other.sqlite.bz2.sha512 release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/b9383abcb72b7a6ba7fa2c68be10649872a649f150a889cc647ca14cd55f5454-filelists.sqlite.bz2 (with props) release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/b9383abcb72b7a6ba7fa2c68be10649872a649f150a889cc647ca14cd55f5454-filelists.sqlite.bz2.asc release/incubator/pulsar/pulsar-2.1.1-incubating/RPMS/repodata/b9383abcb72b7a6ba7fa2c68be10649872a649f150a889cc647ca14cd55f5454-filelists.sqlite.bz2.sha1
[GitHub] srkukarni commented on issue #2593: Add support for running python functions with wheel file
srkukarni commented on issue #2593: Add support for running python functions with wheel file URL: https://github.com/apache/incubator-pulsar/pull/2593#issuecomment-422942590 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kramasamy commented on a change in pull request #2609: Initial commit for a go based pulsar cli.
kramasamy commented on a change in pull request #2609: Initial commit for a go based pulsar cli. URL: https://github.com/apache/incubator-pulsar/pull/2609#discussion_r218946596 ## File path: cli/admin/clusters.go ## @@ -0,0 +1,62 @@ +// Review comment: since the cluster create/delete/modify is used mainly for replication - it might be good to create a replication command group and put these underneath. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2593: Add support for running python functions with wheel file
srkukarni commented on issue #2593: Add support for running python functions with wheel file URL: https://github.com/apache/incubator-pulsar/pull/2593#issuecomment-422936960 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly opened a new pull request #2612: Allow byte[] keys for messages (#1016)
ivankelly opened a new pull request #2612: Allow byte[] keys for messages (#1016) URL: https://github.com/apache/incubator-pulsar/pull/2612 Sometimes it can be useful to send something more complex than a string as the key of the message. However, early on Pulsar chose to make String the only way to send a key, and this permeates throughout the code, so we can't very well change it now. This patch adds rudimentary byte[] key support. If a user adds a byte[] key, the byte[] is base64 encoded and stored in the normal key field. We also send a flag to denote that it is base64 encoded, so the receiving end knows to decode it correctly. There's no schema or anything attached to this. Any SerDe has to be handled manually by the client. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new issue #1016: Support bytes message key
sijie opened a new issue #1016: Support bytes message key URL: https://github.com/apache/incubator-pulsar/issues/1016 Currently the message key is string. It would be good to make the key also be bytes backed. so people can configure a SerDe on the message key, which message router can use to compute which partitions a message should be routed to. It should be considered together with schema registry. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2611: Make dockerUtils use container name for test exec
aahmed-se commented on issue #2611: Make dockerUtils use container name for test exec URL: https://github.com/apache/incubator-pulsar/pull/2611#issuecomment-422930867 We are overriding that , we are reducing the number of log line elements since we are not looking at multithreaded server components but single threaded test instances, also we are making the log files easier to follow bu using the containername This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #2611: Make dockerUtils use container name for test exec
sijie commented on a change in pull request #2611: Make dockerUtils use container name for test exec URL: https://github.com/apache/incubator-pulsar/pull/2611#discussion_r218935033 ## File path: tests/integration/src/test/resources/log4j2.xml ## @@ -0,0 +1,33 @@ + Review comment: why do we need this? the logging is already configured - https://github.com/apache/incubator-pulsar/blob/master/buildtools/src/main/resources/log4j2.xml you can enable logging to console by specifying `-DredirectTestOutputToFile=false` https://github.com/apache/incubator-pulsar/blob/master/pom.xml#L132 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se opened a new pull request #2611: Make dockerUtils use container name for test exec
aahmed-se opened a new pull request #2611: Make dockerUtils use container name for test exec URL: https://github.com/apache/incubator-pulsar/pull/2611 Provides easier to follow logs also we reduce the log output via custom log config for the itegration tests. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2593: Add support for running python functions with wheel file
srkukarni commented on issue #2593: Add support for running python functions with wheel file URL: https://github.com/apache/incubator-pulsar/pull/2593#issuecomment-422919735 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2605: implement topic routing on a per record basis
jerrypeng commented on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422915657 @rdhabalia I will write some tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2607: Add default value for test image org tag
aahmed-se commented on issue #2607: Add default value for test image org tag URL: https://github.com/apache/incubator-pulsar/pull/2607#issuecomment-422913594 you are right will close this This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se closed pull request #2607: Add default value for test image org tag
aahmed-se closed pull request #2607: Add default value for test image org tag URL: https://github.com/apache/incubator-pulsar/pull/2607 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/tests/docker-images/latest-version-image/pom.xml b/tests/docker-images/latest-version-image/pom.xml index b58530ce44..d484972a9a 100644 --- a/tests/docker-images/latest-version-image/pom.xml +++ b/tests/docker-images/latest-version-image/pom.xml @@ -29,6 +29,9 @@ org.apache.pulsar.tests latest-version-image Apache Pulsar :: Tests :: Docker Images :: Latest Version Testing + +apachepulsar + pom This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat opened a new pull request #2610: Prepare website for future releases without '-incubating' suffix
merlimat opened a new pull request #2610: Prepare website for future releases without '-incubating' suffix URL: https://github.com/apache/incubator-pulsar/pull/2610 ### Motivation The `-incubating` suffix is a property of a particular release. We shouldn't add it to all releases in the download page. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2609: Initial commit for a go based pulsar cli.
sijie commented on issue #2609: Initial commit for a go based pulsar cli. URL: https://github.com/apache/incubator-pulsar/pull/2609#issuecomment-422902467 @cckellogg ideally yes. but `mvn license:format` should already attaching the license headers. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] cckellogg commented on issue #2609: Initial commit for a go based pulsar cli.
cckellogg commented on issue #2609: Initial commit for a go based pulsar cli. URL: https://github.com/apache/incubator-pulsar/pull/2609#issuecomment-422901452 @sijie does a license need to be added to the source files only? Do the mod files and readme need a license too? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng removed a comment on issue #2605: implement topic routing on a per record basis
jerrypeng removed a comment on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422897646 rerun tests please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2605: implement topic routing on a per record basis
jerrypeng commented on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422898762 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2593: Add support for running python functions with wheel file
srkukarni commented on issue #2593: Add support for running python functions with wheel file URL: https://github.com/apache/incubator-pulsar/pull/2593#issuecomment-422898627 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2605: implement topic routing on a per record basis
jerrypeng commented on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422897997 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2605: implement topic routing on a per record basis
jerrypeng commented on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422897646 rerun tests please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2605: implement topic routing on a per record basis
jerrypeng commented on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422897747 rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng removed a comment on issue #2605: implement topic routing on a per record basis
jerrypeng removed a comment on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422897519 rerun tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2605: implement topic routing on a per record basis
jerrypeng commented on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422897519 rerun tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on a change in pull request #2605: implement topic routing on a per record basis
jerrypeng commented on a change in pull request #2605: implement topic routing
on a per record basis
URL: https://github.com/apache/incubator-pulsar/pull/2605#discussion_r218900232
##
File path:
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
##
@@ -60,140 +62,153 @@
private final String fqfn;
private interface PulsarSinkProcessor {
-void initializeOutputProducer(String outputTopic, Schema schema,
String fqfn) throws Exception;
TypedMessageBuilder newMessage(Record record) throws Exception;
void sendOutputMessage(TypedMessageBuilder msg, Record record)
throws Exception;
-abstract void close() throws Exception;
+void close() throws Exception;
}
-private class PulsarSinkAtMostOnceProcessor implements
PulsarSinkProcessor {
-private Producer producer;
+private abstract class PulsarSinkProcessorBase implements
PulsarSinkProcessor {
+protected Map> publishProducers = new
ConcurrentHashMap<>();
+protected Schema schema;
-@Override
-public void initializeOutputProducer(String outputTopic, Schema
schema, String fqfn) throws Exception {
-this.producer = AbstractOneOuputTopicProducers.createProducer(
-client, pulsarSinkConfig.getTopic(), null, schema, fqfn);
+protected PulsarSinkProcessorBase(Schema schema) {
+this.schema = schema;
}
-@Override
-public TypedMessageBuilder newMessage(Record record) {
-return producer.newMessage();
+public Producer createProducer(PulsarClient client, String
topic, String producerName, Schema schema, String fqfn)
+throws PulsarClientException {
+ProducerBuilder builder = client.newProducer(schema)
+.blockIfQueueFull(true)
+.enableBatching(true)
+.batchingMaxPublishDelay(1, TimeUnit.MILLISECONDS)
+.compressionType(CompressionType.LZ4)
+.hashingScheme(HashingScheme.Murmur3_32Hash) //
+.messageRoutingMode(MessageRoutingMode.CustomPartition)
+.messageRouter(FunctionResultRouter.of())
+.topic(topic);
+if (producerName != null) {
+builder.producerName(producerName);
+}
+
+return builder
+.property("application", "pulsarfunction")
+.property("fqfn", fqfn).create();
}
-@Override
-public void sendOutputMessage(TypedMessageBuilder msg, Record
record) throws Exception {
-msg.sendAsync();
+protected Producer getProducer(String destinationTopic) {
+return getProducer(destinationTopic, null, destinationTopic);
}
-@Override
-public void close() throws Exception {
-if (null != producer) {
+protected Producer getProducer(String producerId, String
producerName, String topicName) {
+
+Producer producer = publishProducers.get(producerId);
+
+if (producer == null) {
try {
-producer.close();
+Producer newProducer = createProducer(
+client,
+topicName,
+producerName,
+schema,
+fqfn);
+
+Producer existingProducer =
publishProducers.putIfAbsent(producerId, newProducer);
Review comment:
yes because a message can have any destination topic. At the time of
processing, we may or maybe not have already created a producer for that topic.
We cache producers, so that we don't create a producer for each message and
try to reuse existing ones. This is basically the same logic for
MultiConsumerProducer used for effectively once where we have to have separate
producers for each partition
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] srkukarni commented on issue #2593: Add support for running python functions with wheel file
srkukarni commented on issue #2593: Add support for running python functions with wheel file URL: https://github.com/apache/incubator-pulsar/pull/2593#issuecomment-422895917 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on a change in pull request #2605: implement topic routing on a per record basis
jerrypeng commented on a change in pull request #2605: implement topic routing
on a per record basis
URL: https://github.com/apache/incubator-pulsar/pull/2605#discussion_r218900232
##
File path:
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java
##
@@ -60,140 +62,153 @@
private final String fqfn;
private interface PulsarSinkProcessor {
-void initializeOutputProducer(String outputTopic, Schema schema,
String fqfn) throws Exception;
TypedMessageBuilder newMessage(Record record) throws Exception;
void sendOutputMessage(TypedMessageBuilder msg, Record record)
throws Exception;
-abstract void close() throws Exception;
+void close() throws Exception;
}
-private class PulsarSinkAtMostOnceProcessor implements
PulsarSinkProcessor {
-private Producer producer;
+private abstract class PulsarSinkProcessorBase implements
PulsarSinkProcessor {
+protected Map> publishProducers = new
ConcurrentHashMap<>();
+protected Schema schema;
-@Override
-public void initializeOutputProducer(String outputTopic, Schema
schema, String fqfn) throws Exception {
-this.producer = AbstractOneOuputTopicProducers.createProducer(
-client, pulsarSinkConfig.getTopic(), null, schema, fqfn);
+protected PulsarSinkProcessorBase(Schema schema) {
+this.schema = schema;
}
-@Override
-public TypedMessageBuilder newMessage(Record record) {
-return producer.newMessage();
+public Producer createProducer(PulsarClient client, String
topic, String producerName, Schema schema, String fqfn)
+throws PulsarClientException {
+ProducerBuilder builder = client.newProducer(schema)
+.blockIfQueueFull(true)
+.enableBatching(true)
+.batchingMaxPublishDelay(1, TimeUnit.MILLISECONDS)
+.compressionType(CompressionType.LZ4)
+.hashingScheme(HashingScheme.Murmur3_32Hash) //
+.messageRoutingMode(MessageRoutingMode.CustomPartition)
+.messageRouter(FunctionResultRouter.of())
+.topic(topic);
+if (producerName != null) {
+builder.producerName(producerName);
+}
+
+return builder
+.property("application", "pulsarfunction")
+.property("fqfn", fqfn).create();
}
-@Override
-public void sendOutputMessage(TypedMessageBuilder msg, Record
record) throws Exception {
-msg.sendAsync();
+protected Producer getProducer(String destinationTopic) {
+return getProducer(destinationTopic, null, destinationTopic);
}
-@Override
-public void close() throws Exception {
-if (null != producer) {
+protected Producer getProducer(String producerId, String
producerName, String topicName) {
+
+Producer producer = publishProducers.get(producerId);
+
+if (producer == null) {
try {
-producer.close();
+Producer newProducer = createProducer(
+client,
+topicName,
+producerName,
+schema,
+fqfn);
+
+Producer existingProducer =
publishProducers.putIfAbsent(producerId, newProducer);
Review comment:
yes because a message can have a any destination topic. At the time of
processing, we may or maybe not have already created a producer for that topic.
We cache producers, so that we don't create a producer for each message and
try to reuse existing ones. This is basically the same logic for
MultiConsumerProducer used for effectively once where we have to have separate
producers for each partition
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] jerrypeng edited a comment on issue #2605: implement topic routing on a per record basis
jerrypeng edited a comment on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422892352 @rdhabalia a usecase we have is will a source for ingesting data from databases. Users will often want to just specify schema or db and have the data of all the tables underneath automatically be sent to corresponding topics in Pulsar. Thus, at the end of the day we will have table -> topic mapping. For a use case as such, when its too cumbersome to explicitly specify some sort of mapping to topics and also for use cases when you don't know which topic to sent until runtime or upon message inspection This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2605: implement topic routing on a per record basis
jerrypeng commented on issue #2605: implement topic routing on a per record basis URL: https://github.com/apache/incubator-pulsar/pull/2605#issuecomment-422892352 @rdhabalia a use we have is will a source for ingesting data from databases. Users will often want to just specify schema or db and have the data of all the tables underneath automatically be sent to corresponding topics in Pulsar. Thus, at the end of the day we will have table -> topic mapping. For a use case as such, when its too cumbersome to explicitly specify some sort of mapping to topics and also for use cases when you don't know which topic to sent until runtime or upon message inspection This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat closed pull request #2580: [Python] Consolidated duplicated subscribe_*() methods into a single one
merlimat closed pull request #2580: [Python] Consolidated duplicated
subscribe_*() methods into a single one
URL: https://github.com/apache/incubator-pulsar/pull/2580
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/pulsar-client-cpp/python/pulsar/__init__.py
b/pulsar-client-cpp/python/pulsar/__init__.py
index 806c7e2032..6849ecc245 100644
--- a/pulsar-client-cpp/python/pulsar/__init__.py
+++ b/pulsar-client-cpp/python/pulsar/__init__.py
@@ -104,6 +104,9 @@ def send_callback(res, msg):
from pulsar.functions.context import Context
from pulsar.functions.serde import SerDe, IdentitySerDe, PickleSerDe
+import re
+_retype = type(re.compile('x'))
+
class MessageId:
"""
Represents a message id
@@ -412,114 +415,19 @@ def subscribe(self, topic, subscription_name,
unacked_messages_timeout_ms=None,
broker_consumer_stats_cache_time_ms=3,
is_read_compacted=False,
- properties=None
+ properties=None,
+ pattern_auto_discovery_period=60
):
"""
Subscribe to the given topic and subscription combination.
**Args**
-* `topic`: The name of the topic.
-* `subscription`: The name of the subscription.
-
-**Options**
-
-* `consumer_type`:
- Select the subscription type to be used when subscribing to the
topic.
-* `message_listener`:
- Sets a message listener for the consumer. When the listener is set,
- the application will receive messages through it. Calls to
- `consumer.receive()` will not be allowed. The listener function needs
- to accept (consumer, message), for example:
-
-#!python
-def my_listener(consumer, message):
-# process message
-consumer.acknowledge(message)
-
-* `receiver_queue_size`:
- Sets the size of the consumer receive queue. The consumer receive
- queue controls how many messages can be accumulated by the consumer
- before the application calls `receive()`. Using a higher value could
- potentially increase the consumer throughput at the expense of higher
- memory utilization. Setting the consumer queue size to zero decreases
- the throughput of the consumer by disabling pre-fetching of messages.
- This approach improves the message distribution on shared
subscription
- by pushing messages only to those consumers that are ready to process
- them. Neither receive with timeout nor partitioned topics can be used
- if the consumer queue size is zero. The `receive()` function call
- should not be interrupted when the consumer queue size is zero. The
- default value is 1000 messages and should work well for most use
- cases.
-* `max_total_receiver_queue_size_across_partitions`
- Set the max total receiver queue size across partitions.
- This setting will be used to reduce the receiver queue size for
individual partitions
-* `consumer_name`:
- Sets the consumer name.
-* `unacked_messages_timeout_ms`:
- Sets the timeout in milliseconds for unacknowledged messages. The
- timeout needs to be greater than 10 seconds. An exception is thrown
if
- the given value is less than 10 seconds. If a successful
- acknowledgement is not sent within the timeout, all the
unacknowledged
- messages are redelivered.
-* `broker_consumer_stats_cache_time_ms`:
- Sets the time duration for which the broker-side consumer stats will
- be cached in the client.
-* `properties`:
- Sets the properties for the consumer. The properties associated with
a consumer
- can be used for identify a consumer at broker side.
-"""
-_check_type(str, topic, 'topic')
-_check_type(str, subscription_name, 'subscription_name')
-_check_type(ConsumerType, consumer_type, 'consumer_type')
-_check_type(int, receiver_queue_size, 'receiver_queue_size')
-_check_type(int, max_total_receiver_queue_size_across_partitions,
-'max_total_receiver_queue_size_across_partitions')
-_check_type_or_none(str, consumer_name, 'consumer_name')
-_check_type_or_none(int, unacked_messages_timeout_ms,
'unacked_messages_timeout_ms')
-_check_type(int, broker_consumer_stats_cache_time_ms,
'broker_consumer_stats_cache_time_ms')
-_check_type(bool, is_read_compacted, 'is_read_compacted')
-_check_type_or_none(dict, properties, 'properties')
-
-
[incubator-pulsar] branch master updated: [Python] Consolidated duplicated subscribe_*() methods into a single one (#2580)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 99ad32b [Python] Consolidated duplicated subscribe_*() methods into a
single one (#2580)
99ad32b is described below
commit 99ad32b68565e8ee25049741336e5f58571b078a
Author: Matteo Merli
AuthorDate: Wed Sep 19 10:14:59 2018 -0700
[Python] Consolidated duplicated subscribe_*() methods into a single one
(#2580)
* [Python] Consolidated duplicated subscribe_*() methods into a single one
* Fixed missing parameter pattern_auto_discovery_period
---
pulsar-client-cpp/python/pulsar/__init__.py | 229 +++-
pulsar-client-cpp/python/pulsar_test.py | 26 ++--
2 files changed, 35 insertions(+), 220 deletions(-)
diff --git a/pulsar-client-cpp/python/pulsar/__init__.py
b/pulsar-client-cpp/python/pulsar/__init__.py
index 806c7e2..6849ecc 100644
--- a/pulsar-client-cpp/python/pulsar/__init__.py
+++ b/pulsar-client-cpp/python/pulsar/__init__.py
@@ -104,6 +104,9 @@ from pulsar.functions.function import Function
from pulsar.functions.context import Context
from pulsar.functions.serde import SerDe, IdentitySerDe, PickleSerDe
+import re
+_retype = type(re.compile('x'))
+
class MessageId:
"""
Represents a message id
@@ -412,114 +415,19 @@ class Client:
unacked_messages_timeout_ms=None,
broker_consumer_stats_cache_time_ms=3,
is_read_compacted=False,
- properties=None
+ properties=None,
+ pattern_auto_discovery_period=60
):
"""
Subscribe to the given topic and subscription combination.
**Args**
-* `topic`: The name of the topic.
-* `subscription`: The name of the subscription.
-
-**Options**
-
-* `consumer_type`:
- Select the subscription type to be used when subscribing to the
topic.
-* `message_listener`:
- Sets a message listener for the consumer. When the listener is set,
- the application will receive messages through it. Calls to
- `consumer.receive()` will not be allowed. The listener function needs
- to accept (consumer, message), for example:
-
-#!python
-def my_listener(consumer, message):
-# process message
-consumer.acknowledge(message)
-
-* `receiver_queue_size`:
- Sets the size of the consumer receive queue. The consumer receive
- queue controls how many messages can be accumulated by the consumer
- before the application calls `receive()`. Using a higher value could
- potentially increase the consumer throughput at the expense of higher
- memory utilization. Setting the consumer queue size to zero decreases
- the throughput of the consumer by disabling pre-fetching of messages.
- This approach improves the message distribution on shared
subscription
- by pushing messages only to those consumers that are ready to process
- them. Neither receive with timeout nor partitioned topics can be used
- if the consumer queue size is zero. The `receive()` function call
- should not be interrupted when the consumer queue size is zero. The
- default value is 1000 messages and should work well for most use
- cases.
-* `max_total_receiver_queue_size_across_partitions`
- Set the max total receiver queue size across partitions.
- This setting will be used to reduce the receiver queue size for
individual partitions
-* `consumer_name`:
- Sets the consumer name.
-* `unacked_messages_timeout_ms`:
- Sets the timeout in milliseconds for unacknowledged messages. The
- timeout needs to be greater than 10 seconds. An exception is thrown
if
- the given value is less than 10 seconds. If a successful
- acknowledgement is not sent within the timeout, all the
unacknowledged
- messages are redelivered.
-* `broker_consumer_stats_cache_time_ms`:
- Sets the time duration for which the broker-side consumer stats will
- be cached in the client.
-* `properties`:
- Sets the properties for the consumer. The properties associated with
a consumer
- can be used for identify a consumer at broker side.
-"""
-_check_type(str, topic, 'topic')
-_check_type(str, subscription_name, 'subscription_name')
-_check_type(ConsumerType, consumer_type, 'consumer_type')
-_check_type(int, receiver_queue_size, 'receiver_queue_size')
-_check_type(int, max_total_receiver_queue_size_across_partitions,
-
[GitHub] sijie commented on issue #2585: [tests] Make BrokerClientIntegrationTest testing behavior deterministic
sijie commented on issue #2585: [tests] Make BrokerClientIntegrationTest testing behavior deterministic URL: https://github.com/apache/incubator-pulsar/pull/2585#issuecomment-422882486 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2590: Issue #2584: unacked message is not redelivered on time
sijie commented on issue #2590: Issue #2584: unacked message is not redelivered on time URL: https://github.com/apache/incubator-pulsar/pull/2590#issuecomment-422882307 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Correcting SQL getting started title (#2594)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new e357253 Correcting SQL getting started title (#2594) e357253 is described below commit e3572530f904d3cb0b3e80c177dbf33b1f4b2903 Author: Boyang Jerry Peng AuthorDate: Wed Sep 19 10:06:47 2018 -0700 Correcting SQL getting started title (#2594) --- .../versioned_docs/version-2.1.1-incubating/sql-getting-started.md| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/sql-getting-started.md b/site2/website/versioned_docs/version-2.1.1-incubating/sql-getting-started.md index aec0934..3fb42a3 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/sql-getting-started.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/sql-getting-started.md @@ -1,7 +1,7 @@ --- id: version-2.1.1-incubating-sql-getting-started -title: Pulsar SQL Overview -sidebar_label: Overview +title: Pulsar SQL Getting Started +sidebar_label: Getting Started original_id: sql-getting-started ---
[incubator-pulsar] branch jerrypeng-patch-1 deleted (was 6d8b8ba)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a change to branch jerrypeng-patch-1 in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git. was 6d8b8ba Merge remote-tracking branch 'apache/jerrypeng-patch-1' into jerrypeng-patch-1 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[GitHub] sijie closed pull request #2594: Correcting SQL getting started title
sijie closed pull request #2594: Correcting SQL getting started title URL: https://github.com/apache/incubator-pulsar/pull/2594 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Add support for dead letter topics for java functions (#2606)
This is an automated email from the ASF dual-hosted git repository.
sanjeevrk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 82aa2b8 Add support for dead letter topics for java functions (#2606)
82aa2b8 is described below
commit 82aa2b83359c31f71eae40bb8f068ce703f08b59
Author: Sanjeev Kulkarni
AuthorDate: Wed Sep 19 10:01:21 2018 -0700
Add support for dead letter topics for java functions (#2606)
* Added ability to specify dead letter topic to functions
* Fix bug
* Added an example function that fails on a particular message consistently
* Revert change
---
.../org/apache/pulsar/admin/cli/CmdFunctions.java | 25 --
.../functions/instance/JavaInstanceRunnable.java | 5 +++
.../pulsar/functions/source/PulsarSource.java | 18 +-
.../functions/source/PulsarSourceConfig.java | 2 ++
.../api/examples/ConsistentlyFailingFunction.java | 38 ++
.../proto/src/main/proto/Function.proto| 6
.../pulsar/functions/utils/FunctionConfig.java | 2 ++
.../functions/utils/validation/ValidatorImpls.java | 12 +++
8 files changed, 97 insertions(+), 11 deletions(-)
diff --git
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
index e8c8740..5284b59 100644
---
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
+++
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
@@ -24,7 +24,7 @@ import static java.util.Objects.isNull;
import static org.apache.bookkeeper.common.concurrent.FutureUtils.result;
import static org.apache.commons.lang.StringUtils.isBlank;
import static org.apache.commons.lang.StringUtils.isNotBlank;
-import static org.apache.commons.lang3.StringUtils.isNotBlank;
+import static org.apache.commons.lang.StringUtils.isNotEmpty;
import static org.apache.pulsar.common.naming.TopicName.DEFAULT_NAMESPACE;
import static org.apache.pulsar.common.naming.TopicName.PUBLIC_TENANT;
import static org.apache.pulsar.functions.utils.Utils.fileExists;
@@ -45,7 +45,6 @@ import io.netty.buffer.ByteBufUtil;
import io.netty.buffer.Unpooled;
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
import java.lang.reflect.Type;
import java.net.MalformedURLException;
@@ -79,12 +78,12 @@ import org.apache.pulsar.functions.instance.InstanceConfig;
import org.apache.pulsar.functions.proto.Function.ConsumerSpec;
import org.apache.pulsar.functions.proto.Function.FunctionDetails;
import org.apache.pulsar.functions.proto.Function.Resources;
+import org.apache.pulsar.functions.proto.Function.RetryDetails;
import org.apache.pulsar.functions.proto.Function.SinkSpec;
import org.apache.pulsar.functions.proto.Function.SourceSpec;
import org.apache.pulsar.functions.proto.Function.SubscriptionType;
import org.apache.pulsar.functions.runtime.ProcessRuntimeFactory;
import org.apache.pulsar.functions.runtime.RuntimeSpawner;
-import org.apache.pulsar.functions.utils.ConsumerConfig;
import org.apache.pulsar.functions.utils.FunctionConfig;
import org.apache.pulsar.functions.utils.Reflections;
import org.apache.pulsar.functions.utils.Utils;
@@ -321,6 +320,10 @@ public class CmdFunctions extends CmdBase {
protected Long DEPRECATED_timeoutMs;
@Parameter(names = "--timeout-ms", description = "The message timeout
in milliseconds")
protected Long timeoutMs;
+@Parameter(names = "--max-message-retries", description = "How many
times should we try to process a message before giving up")
+protected Integer maxMessageRetries = -1;
+@Parameter(names = "--dead-letter-topic", description = "The topic
where all messages which could not be processed successfully are sent")
+protected String deadLetterTopic;
protected FunctionConfig functionConfig;
protected String userCodeFile;
@@ -464,6 +467,13 @@ public class CmdFunctions extends CmdBase {
functionConfig.setAutoAck(autoAck);
+if (null != maxMessageRetries) {
+functionConfig.setMaxMessageRetries(maxMessageRetries);
+}
+if (null != deadLetterTopic) {
+functionConfig.setDeadLetterTopic(deadLetterTopic);
+}
+
if (null != jarFile) {
functionConfig.setJar(jarFile);
}
@@ -717,6 +727,15 @@ public class CmdFunctions extends CmdBase {
Utils.convertProcessingGuarantee(functionConfig.getProcessingGuarantees()));
}
+if (functionConfig.getMaxMessageRetries() >= 0) {
+RetryDetails.Builder retryBuilder = RetryDetails.newBuilder();
+
[GitHub] srkukarni closed pull request #2606: Add support for dead letter topics for java functions
srkukarni closed pull request #2606: Add support for dead letter topics for
java functions
URL: https://github.com/apache/incubator-pulsar/pull/2606
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
index e8c874059c..5284b59f5b 100644
---
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
+++
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
@@ -24,7 +24,7 @@
import static org.apache.bookkeeper.common.concurrent.FutureUtils.result;
import static org.apache.commons.lang.StringUtils.isBlank;
import static org.apache.commons.lang.StringUtils.isNotBlank;
-import static org.apache.commons.lang3.StringUtils.isNotBlank;
+import static org.apache.commons.lang.StringUtils.isNotEmpty;
import static org.apache.pulsar.common.naming.TopicName.DEFAULT_NAMESPACE;
import static org.apache.pulsar.common.naming.TopicName.PUBLIC_TENANT;
import static org.apache.pulsar.functions.utils.Utils.fileExists;
@@ -45,7 +45,6 @@
import io.netty.buffer.Unpooled;
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
import java.lang.reflect.Type;
import java.net.MalformedURLException;
@@ -79,12 +78,12 @@
import org.apache.pulsar.functions.proto.Function.ConsumerSpec;
import org.apache.pulsar.functions.proto.Function.FunctionDetails;
import org.apache.pulsar.functions.proto.Function.Resources;
+import org.apache.pulsar.functions.proto.Function.RetryDetails;
import org.apache.pulsar.functions.proto.Function.SinkSpec;
import org.apache.pulsar.functions.proto.Function.SourceSpec;
import org.apache.pulsar.functions.proto.Function.SubscriptionType;
import org.apache.pulsar.functions.runtime.ProcessRuntimeFactory;
import org.apache.pulsar.functions.runtime.RuntimeSpawner;
-import org.apache.pulsar.functions.utils.ConsumerConfig;
import org.apache.pulsar.functions.utils.FunctionConfig;
import org.apache.pulsar.functions.utils.Reflections;
import org.apache.pulsar.functions.utils.Utils;
@@ -321,6 +320,10 @@ void processArguments() throws Exception {
protected Long DEPRECATED_timeoutMs;
@Parameter(names = "--timeout-ms", description = "The message timeout
in milliseconds")
protected Long timeoutMs;
+@Parameter(names = "--max-message-retries", description = "How many
times should we try to process a message before giving up")
+protected Integer maxMessageRetries = -1;
+@Parameter(names = "--dead-letter-topic", description = "The topic
where all messages which could not be processed successfully are sent")
+protected String deadLetterTopic;
protected FunctionConfig functionConfig;
protected String userCodeFile;
@@ -464,6 +467,13 @@ void processArguments() throws Exception {
functionConfig.setAutoAck(autoAck);
+if (null != maxMessageRetries) {
+functionConfig.setMaxMessageRetries(maxMessageRetries);
+}
+if (null != deadLetterTopic) {
+functionConfig.setDeadLetterTopic(deadLetterTopic);
+}
+
if (null != jarFile) {
functionConfig.setJar(jarFile);
}
@@ -717,6 +727,15 @@ protected FunctionDetails convert(FunctionConfig
functionConfig)
Utils.convertProcessingGuarantee(functionConfig.getProcessingGuarantees()));
}
+if (functionConfig.getMaxMessageRetries() >= 0) {
+RetryDetails.Builder retryBuilder = RetryDetails.newBuilder();
+
retryBuilder.setMaxMessageRetries(functionConfig.getMaxMessageRetries());
+if (isNotEmpty(functionConfig.getDeadLetterTopic())) {
+
retryBuilder.setDeadLetterTopic(functionConfig.getDeadLetterTopic());
+}
+functionDetailsBuilder.setRetryDetails(retryBuilder);
+}
+
Map configs = new HashMap<>();
configs.putAll(functionConfig.getUserConfig());
diff --git
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
index b3f86eae0e..1e07516f59 100644
---
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
+++
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
@@ -547,6 +547,11 @@ public void setupInput(ContextImpl contextImpl) throws
Exception {
[GitHub] sijie commented on issue #2606: Add support for dead letter topics for java functions
sijie commented on issue #2606: Add support for dead letter topics for java functions URL: https://github.com/apache/incubator-pulsar/pull/2606#issuecomment-422843947 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2594: Correcting SQL getting started title
sijie commented on issue #2594: Correcting SQL getting started title URL: https://github.com/apache/incubator-pulsar/pull/2594#issuecomment-422843819 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam removed a comment on issue #2572: Shorten the timeout value of C++ ZTS client
massakam removed a comment on issue #2572: Shorten the timeout value of C++ ZTS client URL: https://github.com/apache/incubator-pulsar/pull/2572#issuecomment-422764843 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam commented on issue #2572: Shorten the timeout value of C++ ZTS client
massakam commented on issue #2572: Shorten the timeout value of C++ ZTS client URL: https://github.com/apache/incubator-pulsar/pull/2572#issuecomment-422816585 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2593: Add support for running python functions with wheel file
srkukarni commented on issue #2593: Add support for running python functions with wheel file URL: https://github.com/apache/incubator-pulsar/pull/2593#issuecomment-422807920 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2606: Add support for dead letter topics for java functions
srkukarni commented on issue #2606: Add support for dead letter topics for java functions URL: https://github.com/apache/incubator-pulsar/pull/2606#issuecomment-422807563 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam commented on issue #2572: Shorten the timeout value of C++ ZTS client
massakam commented on issue #2572: Shorten the timeout value of C++ ZTS client URL: https://github.com/apache/incubator-pulsar/pull/2572#issuecomment-422764843 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch jerrypeng-patch-1 updated (e256a79 -> 6d8b8ba)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a change to branch jerrypeng-patch-1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git.
from e256a79 Correcting SQL getting started title
add e4eb459 Correcting SQL getting started title
add 9e1691e Merge remote-tracking branch 'apache/master' into
jerrypeng-patch-1
add e677679 Merge remote-tracking branch 'apache/master' into
jerrypeng-patch-1
add 49fc5e5 [Function] avoid creating assignment snapshot and publish
individual assigment msg (#2549)
add 701f3ea fix behavior of JSONSchema for derived classes (#2577)
add 3097283 docs for 2.1.1-incubating release (#2601)
add 8e14175 add auto ack and timeout configurable (#2503)
add 4fdc21a [tests] remove
PersistentQueueE2ETest#testConsumersWithDifferentPermits (#2604)
add 5bdfa8a Merge remote-tracking branch 'apache/master' into
jerrypeng-patch-1
add 6d8b8ba Merge remote-tracking branch 'apache/jerrypeng-patch-1' into
jerrypeng-patch-1
No new revisions were added by this update.
Summary of changes:
conf/functions_worker.yml | 2 +
.../pulsar/compaction/TwoPhaseCompactor.java | 2 +-
.../apache/pulsar/compaction/CompactionTest.java | 2 +-
.../worker/PulsarWorkerAssignmentTest.java | 370 +
pulsar-client-schema/pom.xml | 6 +
.../pulsar/client/impl/schema/JSONSchema.java | 41 ++-
.../pulsar/client/schema/JSONSchemaTest.java | 67 +++-
.../pulsar/client/schema/SchemaTestUtils.java | 42 ++-
.../java/org/apache/pulsar/admin/cli/CmdSinks.java | 19 +-
.../org/apache/pulsar/admin/cli/TestCmdSinks.java | 1 +
.../proto/src/main/proto/Request.proto | 5 -
.../apache/pulsar/functions/utils/SinkConfig.java | 3 +
.../functions/worker/FunctionAssignmentTailer.java | 47 ++-
.../functions/worker/FunctionRuntimeManager.java | 283 +++-
.../pulsar/functions/worker/SchedulerManager.java | 126 ---
.../pulsar/functions/worker/WorkerConfig.java | 5 +-
.../pulsar/functions/worker/WorkerService.java | 15 +-
.../worker/scheduler/RoundRobinScheduler.java | 10 +-
.../worker/FunctionRuntimeManagerTest.java | 40 +--
.../functions/worker/MembershipManagerTest.java| 3 +
.../functions/worker/SchedulerManagerTest.java | 298 +
site2/docs/reference-pulsar-admin.md | 4 +
site2/website/release-notes.md | 17 +
site2/website/releases.json| 1 +
site2/website/sidebars.json| 5 -
.../version-2.1.1-incubating}/adaptors-kafka.md| 3 +-
.../administration-proxy.md| 3 +-
.../client-libraries-cpp.md| 3 +-
.../client-libraries-python.md | 3 +-
.../cookbooks-tiered-storage.md| 3 +-
.../version-2.1.1-incubating}/deploy-bare-metal.md | 3 +-
.../version-2.1.1-incubating}/functions-api.md | 3 +-
.../functions-deploying.md | 3 +-
.../functions-guarantees.md| 3 +-
.../functions-overview.md | 3 +-
.../functions-quickstart.md| 3 +-
.../getting-started-clients.md | 3 +-
.../getting-started-standalone.md | 3 +-
.../version-2.1.1-incubating}/io-managing.md | 3 +-
.../version-2.1.1-incubating}/io-quickstart.md | 7 +-
.../reference-configuration.md | 3 +-
.../reference-pulsar-admin.md | 3 +-
.../sql-deployment-configurations.md | 3 +-
.../sql-getting-started.md | 3 +-
.../version-2.1.1-incubating}/sql-overview.md | 3 +-
site2/website/versions.json| 1 +
46 files changed, 1016 insertions(+), 463 deletions(-)
create mode 100644
pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarWorkerAssignmentTest.java
copy site2/{docs =>
website/versioned_docs/version-2.1.1-incubating}/adaptors-kafka.md (99%)
copy site2/{docs =>
website/versioned_docs/version-2.1.1-incubating}/administration-proxy.md (98%)
copy site2/{docs =>
website/versioned_docs/version-2.1.1-incubating}/client-libraries-cpp.md (98%)
copy site2/{docs =>
website/versioned_docs/version-2.1.1-incubating}/client-libraries-python.md
(97%)
copy site2/{docs =>
website/versioned_docs/version-2.1.1-incubating}/cookbooks-tiered-storage.md
(99%)
copy site2/{docs =>
website/versioned_docs/version-2.1.1-incubating}/deploy-bare-metal.md (99%)
copy site2/{docs =>
website/versioned_docs/version-2.1.1-incubating}/functions-api.md (99%)
copy site2/{docs =>
website/versioned_docs/version-2.1.1-incubating}/functions-deploying.md (99%)
copy site2/{docs =>
[incubator-pulsar] branch asf-site updated: Updated site at revision ee5afa5
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/asf-site by this push: new af2c2fa Updated site at revision ee5afa5 af2c2fa is described below commit af2c2fabb89b7bf3f5cab770233090ee90819ec3 Author: jenkins AuthorDate: Wed Sep 19 08:53:40 2018 + Updated site at revision ee5afa5 --- content/api/client/allclasses-frame.html | 1 + content/api/client/allclasses-noframe.html | 1 + content/api/client/index-all.html | 30 .../apache/pulsar/client/api/ClientBuilder.html| 25 ++- .../org/apache/pulsar/client/api/PulsarClient.html | 82 ++ .../apache/pulsar/client/api/ReaderListener.html | 4 +- ...ReaderListener.html => ServiceUrlProvider.html} | 77 +--- .../client/api/SubscriptionInitialPosition.html| 4 +- .../apache/pulsar/client/api/package-frame.html| 1 + .../apache/pulsar/client/api/package-summary.html | 9 ++- .../org/apache/pulsar/client/api/package-tree.html | 1 + content/api/client/overview-tree.html | 1 + content/docs/en/next/pulsar-admin.html | 4 ++ content/docs/en/next/pulsar-admin/index.html | 4 ++ content/docs/fr/next/pulsar-admin.html | 4 ++ content/docs/fr/next/pulsar-admin/index.html | 4 ++ content/docs/ja/next/pulsar-admin.html | 4 ++ content/docs/ja/next/pulsar-admin/index.html | 4 ++ .../docs/latest/adaptors/PulsarSpark/index.html| 10 +-- .../docs/latest/adaptors/PulsarStorm/index.html| 8 +-- content/docs/latest/admin-api/overview/index.html | 18 ++--- content/docs/latest/clients/Cpp/index.html | 8 +-- content/docs/latest/clients/Java/index.html| 16 ++--- content/docs/latest/clients/Python/index.html | 10 +-- content/docs/latest/clients/WebSocket/index.html | 8 +-- content/docs/latest/clients/go/index.html | 6 +- .../docs/latest/cookbooks/Encryption/index.html| 6 +- .../latest/cookbooks/PartitionedTopics/index.html | 16 ++--- .../latest/cookbooks/RetentionExpiry/index.html| 16 ++--- .../docs/latest/cookbooks/compaction/index.html| 8 +-- .../cookbooks/message-deduplication/index.html | 12 ++-- .../docs/latest/cookbooks/message-queue/index.html | 16 ++--- .../latest/cookbooks/tiered-storage/index.html | 6 +- .../docs/latest/deployment/Kubernetes/index.html | 4 +- .../docs/latest/deployment/aws-cluster/index.html | 6 +- content/docs/latest/deployment/cluster/index.html | 6 +- content/docs/latest/deployment/instance/index.html | 6 +- .../ConceptsAndArchitecture/index.html | 4 +- .../latest/getting-started/LocalCluster/index.html | 4 +- .../latest/getting-started/Pulsar-2.0/index.html | 2 +- .../docs/latest/getting-started/docker/index.html | 4 +- .../docs/latest/project/BinaryProtocol/index.html | 4 +- content/docs/latest/project/CompileCpp/index.html | 8 +-- .../docs/latest/project/SimulationTools/index.html | 2 +- .../docs/latest/project/schema-storage/index.html | 4 +- content/docs/latest/reference/CliTools/index.html | 22 +++--- .../docs/latest/security/authorization/index.html | 14 ++-- content/docs/latest/security/encryption/index.html | 6 +- content/docs/zh-CN/next/pulsar-admin.html | 4 ++ content/docs/zh-CN/next/pulsar-admin/index.html| 4 ++ content/ja/adaptors/PulsarSpark/index.html | 8 +-- content/ja/adaptors/PulsarStorm/index.html | 6 +- content/ja/admin/AdminInterface/index.html | 12 ++-- content/ja/admin/Authz/index.html | 12 ++-- content/ja/admin/ClustersBrokers/index.html| 6 +- content/ja/admin/PropertiesNamespaces/index.html | 6 +- content/ja/advanced/PartitionedTopics/index.html | 12 ++-- content/ja/advanced/RetentionExpiry/index.html | 12 ++-- content/ja/clients/Cpp/index.html | 6 +- content/ja/clients/Java/index.html | 8 +-- content/ja/clients/Python/index.html | 8 +-- content/ja/clients/WebSocket/index.html| 8 +-- content/ja/deployment/InstanceSetup/index.html | 6 +- content/ja/deployment/Kubernetes/index.html| 4 +- .../ConceptsAndArchitecture/index.html | 2 +- content/ja/getting-started/LocalCluster/index.html | 4 +- content/ja/project/BinaryProtocol/index.html | 4 +- content/ja/project/SimulationTools/index.html | 2 +- content/ja/reference/CliTools/index.html | 18 ++--- content/swagger/swagger.json | 34 - 70 files changed, 424 insertions(+), 282 deletions(-) diff --git a/content/api/client/allclasses-frame.html b/content/api/client/allclasses-frame.html index 3244593..bd10c76 100644 ---
[GitHub] sijie commented on issue #2543: Add ServiceUrlProvider and add method forceCloseConnection in PulsarC…
sijie commented on issue #2543: Add ServiceUrlProvider and add method forceCloseConnection in PulsarC… URL: https://github.com/apache/incubator-pulsar/pull/2543#issuecomment-422704688 @codelipenghui yeah, thank you so much for your contribution! look forward to your contribution about zookeeper based service url provider :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Add ServiceUrlProvider and add method forceCloseConnection in PulsarC… (#2543)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new ee5afa5 Add ServiceUrlProvider and add method forceCloseConnection in
PulsarC… (#2543)
ee5afa5 is described below
commit ee5afa5007e260891140956d244a865d73368321
Author: penghui
AuthorDate: Wed Sep 19 15:44:15 2018 +0800
Add ServiceUrlProvider and add method forceCloseConnection in PulsarC…
(#2543)
Support build Pulsar client with serviceUrlProvider method.
### Motivation
With serviceUrlProvider we can store the pulsar service url in zookeeper or
any other config service.
And we can watch the service url change event then control the pulsar
client, such as change pulsar client serviceUrl, force close client connection
or re-connect with new service url.
### Modifications
Add ServiceUrlProvider interface.
Add forceCloseConnection method in PulsarClient.
---
.../pulsar/client/api/ServiceUrlProviderTest.java | 156 +
.../apache/pulsar/client/api/ClientBuilder.java| 7 +
.../org/apache/pulsar/client/api/PulsarClient.java | 22 +++
.../pulsar/client/api/ServiceUrlProvider.java | 42 ++
.../pulsar/client/impl/ClientBuilderImpl.java | 20 ++-
.../apache/pulsar/client/impl/HandlerState.java| 4 +
.../apache/pulsar/client/impl/LookupService.java | 2 +-
.../client/impl/PartitionedProducerImpl.java | 4 +
.../pulsar/client/impl/PulsarClientImpl.java | 29 +++-
.../client/impl/conf/ClientConfigurationData.java | 2 +
10 files changed, 283 insertions(+), 5 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/ServiceUrlProviderTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/ServiceUrlProviderTest.java
new file mode 100644
index 000..ab86f12
--- /dev/null
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/ServiceUrlProviderTest.java
@@ -0,0 +1,156 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api;
+
+import org.apache.bookkeeper.test.PortManager;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.client.impl.ConsumerImpl;
+import org.apache.pulsar.client.impl.ProducerImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import java.util.concurrent.TimeUnit;
+
+public class ServiceUrlProviderTest extends ProducerConsumerBase {
+
+@BeforeClass
+@Override
+protected void setup() throws Exception {
+super.internalSetup();
+super.producerBaseSetup();
+
+}
+
+@AfterClass
+@Override
+protected void cleanup() throws Exception {
+super.internalCleanup();
+}
+
+@Test
+public void testCreateClientWithServiceUrlProvider() throws Exception {
+
+PulsarClient client = PulsarClient.builder()
+.serviceUrlProvider(new
TestServiceUrlProvider(pulsar.getBrokerServiceUrl()))
+.statsInterval(1, TimeUnit.SECONDS)
+.build();
+Assert.assertTrue(((PulsarClientImpl)
client).getConfiguration().getServiceUrlProvider() instanceof
TestServiceUrlProvider);
+Producer producer = client.newProducer(Schema.STRING)
+.topic("persistent://my-property/my-ns/my-topic")
+.create();
+Consumer consumer = client.newConsumer(Schema.STRING)
+.topic("persistent://my-property/my-ns/my-topic")
+.subscriptionName("my-subscribe")
+.subscribe();
+for (int i = 0; i < 100; i++) {
+producer.send("Hello Pulsar[" + i + "]");
+}
+client.forceCloseConnection();
+for (int i = 100; i < 200; i++) {
+producer.send("Hello Pulsar[" + i + "]");
+}
+int received = 0;
+do {
+Message message = consumer.receive();
+
[GitHub] sijie commented on issue #2585: [tests] Make BrokerClientIntegrationTest testing behavior deterministic
sijie commented on issue #2585: [tests] Make BrokerClientIntegrationTest testing behavior deterministic URL: https://github.com/apache/incubator-pulsar/pull/2585#issuecomment-422678548 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam commented on issue #2572: Shorten the timeout value of C++ ZTS client
massakam commented on issue #2572: Shorten the timeout value of C++ ZTS client URL: https://github.com/apache/incubator-pulsar/pull/2572#issuecomment-422675074 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: fixing/improving logging for function instance (#2586)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 3f8bd00 fixing/improving logging for function instance (#2586) 3f8bd00 is described below commit 3f8bd00d961f1f56233c629746c11cbd711bf06e Author: Boyang Jerry Peng AuthorDate: Tue Sep 18 23:30:39 2018 -0700 fixing/improving logging for function instance (#2586) ### Motivation logging for java function instance (process mode and localrun) is not working well. Log files are created at inappropriate locations for logs not captured by the routing appender, functions bookkeeper logs are missing because we are not using the shaded classpath. The best for logging is just to put all the logs from the java instance log in a single log and not worry about the routing which is only useful when running in threaded mode --- .../pulsar/functions/runtime/JavaInstanceMain.java | 9 --- .../pulsar/functions/runtime/ProcessRuntime.java | 22 +++--- .../pulsar/functions/runtime/ThreadRuntime.java| 4 +- .../src/main/resources/java_instance_log4j2.yml| 78 +++--- .../functions/runtime/ProcessRuntimeTest.java | 4 +- 5 files changed, 31 insertions(+), 86 deletions(-) diff --git a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceMain.java b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceMain.java index 11da7c3..083686b 100644 --- a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceMain.java +++ b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceMain.java @@ -22,27 +22,18 @@ package org.apache.pulsar.functions.runtime; import com.beust.jcommander.JCommander; import com.beust.jcommander.Parameter; import com.beust.jcommander.converters.StringConverter; -import com.google.gson.Gson; import com.google.protobuf.Empty; import com.google.protobuf.util.JsonFormat; import io.grpc.Server; import io.grpc.ServerBuilder; import io.grpc.stub.StreamObserver; import lombok.extern.slf4j.Slf4j; - -import static org.apache.commons.lang3.StringUtils.isNotBlank; - import org.apache.pulsar.functions.instance.AuthenticationConfig; import org.apache.pulsar.functions.instance.InstanceConfig; -import org.apache.pulsar.functions.proto.Function; -import org.apache.pulsar.functions.proto.Function.ProcessingGuarantees; -import org.apache.pulsar.functions.proto.Function.SinkSpec; -import org.apache.pulsar.functions.proto.Function.SourceSpec; import org.apache.pulsar.functions.proto.Function.FunctionDetails; import org.apache.pulsar.functions.proto.InstanceCommunication; import org.apache.pulsar.functions.proto.InstanceControlGrpc; -import java.util.Map; import java.util.TimerTask; import java.util.concurrent.ExecutionException; import java.util.concurrent.Executors; diff --git a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntime.java b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntime.java index a978a09..c5159b0 100644 --- a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntime.java +++ b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntime.java @@ -22,32 +22,32 @@ package org.apache.pulsar.functions.runtime; import com.google.common.util.concurrent.FutureCallback; import com.google.common.util.concurrent.Futures; import com.google.common.util.concurrent.ListenableFuture; -import com.google.gson.Gson; import com.google.protobuf.Empty; import com.google.protobuf.util.JsonFormat; import io.grpc.ManagedChannel; import io.grpc.ManagedChannelBuilder; import lombok.Getter; import lombok.extern.slf4j.Slf4j; - -import static org.apache.commons.lang3.StringUtils.isNotBlank; - -import org.apache.commons.lang3.StringUtils; import org.apache.pulsar.functions.instance.AuthenticationConfig; import org.apache.pulsar.functions.instance.InstanceConfig; import org.apache.pulsar.functions.proto.Function; import org.apache.pulsar.functions.proto.InstanceCommunication; import org.apache.pulsar.functions.proto.InstanceCommunication.FunctionStatus; import org.apache.pulsar.functions.proto.InstanceControlGrpc; +import org.apache.pulsar.functions.utils.FunctionDetailsUtils; import org.apache.pulsar.functions.utils.functioncache.FunctionCacheEntry; import java.io.InputStream; -import java.util.*; +import java.util.LinkedList; +import java.util.List; +import java.util.TimerTask; import java.util.concurrent.CompletableFuture; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; +import static
[GitHub] sijie closed pull request #2586: fixing/improving logging for function instance
sijie closed pull request #2586: fixing/improving logging for function instance
URL: https://github.com/apache/incubator-pulsar/pull/2586
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceMain.java
b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceMain.java
index 11da7c30ac..083686b503 100644
---
a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceMain.java
+++
b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceMain.java
@@ -22,27 +22,18 @@
import com.beust.jcommander.JCommander;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.converters.StringConverter;
-import com.google.gson.Gson;
import com.google.protobuf.Empty;
import com.google.protobuf.util.JsonFormat;
import io.grpc.Server;
import io.grpc.ServerBuilder;
import io.grpc.stub.StreamObserver;
import lombok.extern.slf4j.Slf4j;
-
-import static org.apache.commons.lang3.StringUtils.isNotBlank;
-
import org.apache.pulsar.functions.instance.AuthenticationConfig;
import org.apache.pulsar.functions.instance.InstanceConfig;
-import org.apache.pulsar.functions.proto.Function;
-import org.apache.pulsar.functions.proto.Function.ProcessingGuarantees;
-import org.apache.pulsar.functions.proto.Function.SinkSpec;
-import org.apache.pulsar.functions.proto.Function.SourceSpec;
import org.apache.pulsar.functions.proto.Function.FunctionDetails;
import org.apache.pulsar.functions.proto.InstanceCommunication;
import org.apache.pulsar.functions.proto.InstanceControlGrpc;
-import java.util.Map;
import java.util.TimerTask;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
diff --git
a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntime.java
b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntime.java
index a978a09cb0..c5159b0706 100644
---
a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntime.java
+++
b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntime.java
@@ -22,32 +22,32 @@
import com.google.common.util.concurrent.FutureCallback;
import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
-import com.google.gson.Gson;
import com.google.protobuf.Empty;
import com.google.protobuf.util.JsonFormat;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
-
-import static org.apache.commons.lang3.StringUtils.isNotBlank;
-
-import org.apache.commons.lang3.StringUtils;
import org.apache.pulsar.functions.instance.AuthenticationConfig;
import org.apache.pulsar.functions.instance.InstanceConfig;
import org.apache.pulsar.functions.proto.Function;
import org.apache.pulsar.functions.proto.InstanceCommunication;
import org.apache.pulsar.functions.proto.InstanceCommunication.FunctionStatus;
import org.apache.pulsar.functions.proto.InstanceControlGrpc;
+import org.apache.pulsar.functions.utils.FunctionDetailsUtils;
import org.apache.pulsar.functions.utils.functioncache.FunctionCacheEntry;
import java.io.InputStream;
-import java.util.*;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.TimerTask;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
/**
* A function container implemented using java thread.
*/
@@ -97,8 +97,14 @@
// by the child process and manually added to classpath
args.add(String.format("-D%s=%s",
FunctionCacheEntry.JAVA_INSTANCE_JAR_PROPERTY, instanceFile));
args.add("-Dlog4j.configurationFile=java_instance_log4j2.yml");
-args.add("-Dpulsar.log.dir=" + logDirectory);
-args.add("-Dpulsar.log.file=" +
instanceConfig.getFunctionDetails().getName());
+args.add("-Dpulsar.function.log.dir=" + String.format(
+"%s/%s",
+logDirectory,
+
FunctionDetailsUtils.getFullyQualifiedName(instanceConfig.getFunctionDetails(;
+args.add("-Dpulsar.function.log.file=" + String.format(
+"%s-%s",
+instanceConfig.getFunctionDetails().getName(),
+instanceConfig.getInstanceId()));
if (instanceConfig.getFunctionDetails().getResources() != null) {
[GitHub] jerrypeng commented on issue #2586: fixing/improving logging for function instance
jerrypeng commented on issue #2586: fixing/improving logging for function instance URL: https://github.com/apache/incubator-pulsar/pull/2586#issuecomment-422669848 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2606: Add support for dead letter topics for java functions
srkukarni commented on issue #2606: Add support for dead letter topics for java functions URL: https://github.com/apache/incubator-pulsar/pull/2606#issuecomment-422666303 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
