[GitHub] jerrypeng commented on issue #1833: adding worker to instance health check

2018-05-22 Thread GitBox 
jerrypeng commented on issue #1833: adding worker to instance health check
URL: https://github.com/apache/incubator-pulsar/pull/1833#issuecomment-391221899
 
 
   @srkukarni I have added the check for python as well.  Please review again


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] srkukarni commented on a change in pull request #1833: adding worker to instance health check

2018-05-22 Thread GitBox 
srkukarni commented on a change in pull request #1833: adding worker to 
instance health check
URL: https://github.com/apache/incubator-pulsar/pull/1833#discussion_r190090206
 
 

 ##
 File path: pulsar-functions/proto/src/main/proto/InstanceCommunication.proto
 ##
 @@ -65,7 +65,12 @@ message MetricsData {
 map metrics = 1;
 }
 
+message HealthCheckResult {
 
 Review comment:
   I believe that we periodically already do getFunctionStatus? Do we want to 
overload that function instead of adding a new one?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jerrypeng commented on issue #1833: adding worker to instance health check

2018-05-22 Thread GitBox 
jerrypeng commented on issue #1833: adding worker to instance health check
URL: https://github.com/apache/incubator-pulsar/pull/1833#issuecomment-391187925
 
 
   @srkukarni I have refactored based on our discussion


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bruth commented on issue #1764: WIP - Pulsar Go client library

2018-05-22 Thread GitBox 
bruth commented on issue #1764: WIP - Pulsar Go client library
URL: https://github.com/apache/incubator-pulsar/pull/1764#issuecomment-391183835
 
 
   Great changes.
   
   > This I think is more related to cgo and MacOS. I have the same but it was 
working on linux.
   
   Good to know. I will be deploying on Linux so that works for me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bruth commented on a change in pull request #1764: WIP - Pulsar Go client library

2018-05-22 Thread GitBox 
bruth commented on a change in pull request #1764: WIP - Pulsar Go client 
library
URL: https://github.com/apache/incubator-pulsar/pull/1764#discussion_r190093869
 
 

 ##
 File path: pulsar-client-go/pulsar/logger.go
 ##
 @@ -0,0 +1,48 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package pulsar
+
+import "fmt"
+
+type LoggerLevel int
+
+const (
+   DEBUG LoggerLevel = 0;
+   INFO  LoggerLevel = 1;
+   WARN  LoggerLevel = 2;
+   ERROR LoggerLevel = 3;
 
 Review comment:
   Also no semicolons.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bruth commented on a change in pull request #1764: WIP - Pulsar Go client library

2018-05-22 Thread GitBox 
bruth commented on a change in pull request #1764: WIP - Pulsar Go client 
library
URL: https://github.com/apache/incubator-pulsar/pull/1764#discussion_r190093685
 
 

 ##
 File path: pulsar-client-go/pulsar/logger.go
 ##
 @@ -0,0 +1,48 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package pulsar
+
+import "fmt"
+
+type LoggerLevel int
+
+const (
+   DEBUG LoggerLevel = 0;
+   INFO  LoggerLevel = 1;
+   WARN  LoggerLevel = 2;
+   ERROR LoggerLevel = 3;
+)
+
+
+func (l LoggerLevel) String() string {
+   switch l {
+   case DEBUG:
+   return "DEBUG"
+   case INFO:
+   return "INFO "
+   case WARN:
+   return "WARN "
 
 Review comment:
   Remove trailing spaces for WARN and INFO.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jerrypeng commented on a change in pull request #1833: adding worker to instance health check

2018-05-22 Thread GitBox 
jerrypeng commented on a change in pull request #1833: adding worker to 
instance health check
URL: https://github.com/apache/incubator-pulsar/pull/1833#discussion_r190091037
 
 

 ##
 File path: 
pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ThreadRuntime.java
 ##
 @@ -114,6 +114,18 @@ public void stop() {
 return 
CompletableFuture.completedFuture(javaInstanceRunnable.getAndResetMetrics());
 }
 
+@Override
 
 Review comment:
   can we do this for java first and add to python in subsequent PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1764: WIP - Pulsar Go client library

2018-05-22 Thread GitBox 
merlimat commented on issue #1764: WIP - Pulsar Go client library
URL: https://github.com/apache/incubator-pulsar/pull/1764#issuecomment-391173087
 
 
   >  Rename MessageBuilder to a ProducerMessage since it is no longer 
implements a builder pattern?
   
   Yes, I didn't want to call it `Message` or `MessageOptions`, 
`ProducerMessage` sounds good.
   
   >  It may be useful to include the message in SendAsync error callback in 
order to correlate the error to the message.
   
   Yes, mostly in C++ or Java you attach that by adding the value on the 
lambda. Changed it here
   
   >  There should be control over logging, either turning it off or provide an 
interface to wrap it somehow so the structure can be controlled.
   
   Good point, the log4cxx based logger was a bit overkill and not well 
integrated. Replaced now and allow to set logger function callback.
   
   > I noticed I could not create a binary that statically linked libpulsar. 
This could be my environment, but this was the output:
   
   This I think is more related to cgo and MacOS. I have the same but it was 
working on linux.
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jerrypeng closed pull request #1828: instance worker health check

2018-05-22 Thread GitBox 
jerrypeng closed pull request #1828: instance worker health check
URL: https://github.com/apache/incubator-pulsar/pull/1828
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java 
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java
index 2a0e1a22e9..88cf318247 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java
@@ -146,6 +146,7 @@ private static boolean argsContains(String[] args, String 
arg) {
 // worker talks to local broker
 workerConfig.setPulsarServiceUrl("pulsar://127.0.0.1:" + 
brokerConfig.getBrokerServicePort());
 workerConfig.setPulsarWebServiceUrl("http://127.0.0.1:; + 
brokerConfig.getWebServicePort());
+workerConfig.setWorkerPort(brokerConfig.getWebServicePort());
 String hostname = 
ServiceConfigurationUtils.getDefaultOrConfiguredAddress(
 brokerConfig.getAdvertisedAddress());
 workerConfig.setWorkerHostname(hostname);
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java 
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
index 641636afed..f6ba9f569c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
@@ -189,6 +189,7 @@ void start() throws Exception {
 // worker talks to local broker
 workerConfig.setPulsarServiceUrl("pulsar://127.0.0.1:" + 
config.getBrokerServicePort());
 workerConfig.setPulsarWebServiceUrl("http://127.0.0.1:; + 
config.getWebServicePort());
+workerConfig.setWorkerPort(config.getWebServicePort());
 String hostname = 
ServiceConfigurationUtils.getDefaultOrConfiguredAddress(
 config.getAdvertisedAddress());
 workerConfig.setWorkerHostname(hostname);
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/FunctionsBase.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/FunctionsBase.java
index c22b6117b2..a08b9a8eba 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/FunctionsBase.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/FunctionsBase.java
@@ -221,6 +221,20 @@ public Response getAssignments() {
 return functions.getAssignments();
 }
 
+@GET
+@ApiOperation(
+value = "Fetches unqiue Id of worker",
+response = String.class,
+responseContainer = "String"
+)
+@ApiResponses(value = {
+@ApiResponse(code = 403, message = "The requester doesn't have 
admin permissions")
+})
+@Path("/id")
+public Response getId() {
+return functions.getId();
+}
+
 @POST
 @ApiOperation(
 value = "Triggers a Pulsar Function with a user-specified value or 
file data",
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/InstanceConfig.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/InstanceConfig.java
index 9f9da7986a..4e73eeaf7f 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/InstanceConfig.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/InstanceConfig.java
@@ -41,4 +41,6 @@
 private FunctionDetails functionDetails;
 private int maxBufferedTuples;
 private int port;
+private String fullyQualifiedWorkerId;
+private int workerPort;
 }
diff --git a/pulsar-functions/instance/src/main/python/python_instance_main.py 
b/pulsar-functions/instance/src/main/python/python_instance_main.py
index 0c9c5bf3f9..26e6706778 100644
--- a/pulsar-functions/instance/src/main/python/python_instance_main.py
+++ b/pulsar-functions/instance/src/main/python/python_instance_main.py
@@ -74,6 +74,8 @@ def main():
   parser.add_argument('--source_topics_serde_classname', required=True, 
help='A mapping of Input topics to SerDe')
   parser.add_argument('--sink_topic', required=False, help='Sink Topic')
   parser.add_argument('--sink_serde_classname', required=False, help='Sink 
SerDe classname')
+  parser.add_argument('--fully_qualified_worker_id', required=True, 
help='Unique Identifier for worker')
+  parser.add_argument('--worker_port', required=True, help='Service port of 
worker service')
 
   args = parser.parse_args()
   log_file = os.path.join(args.logging_directory,
diff --git

[incubator-pulsar] branch master updated: Don't set Pulsar Memory if variable is already set (#1831)

2018-05-22 Thread sijie 
This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 928ba3d  Don't set Pulsar Memory if variable is already set (#1831)
928ba3d is described below

commit 928ba3d924c32e9bfc2641fecac35064e0228765
Author: Sanjeev Kulkarni 
AuthorDate: Tue May 22 15:03:32 2018 -0700

Don't set Pulsar Memory if variable is already set (#1831)

* Don't set Pulsar Memory if variable is already set

* Took feedback
---
 conf/pulsar_tools_env.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/conf/pulsar_tools_env.sh b/conf/pulsar_tools_env.sh
index 7eca607..7ad022f 100755
--- a/conf/pulsar_tools_env.sh
+++ b/conf/pulsar_tools_env.sh
@@ -42,7 +42,7 @@
 # PULSAR_GLOBAL_ZK_CONF=
 
 # Extra options to be passed to the jvm
-PULSAR_MEM=" -Xmx256m -XX:MaxDirectMemorySize=256m"
+PULSAR_MEM=${PULSAR_MEM:-"-Xmx256m -XX:MaxDirectMemorySize=256m"}
 
 # Garbage collection options
 PULSAR_GC=" -client "

-- 
To stop receiving notification emails like this one, please contact
si...@apache.org.

[GitHub] sijie closed pull request #1831: Don't set Pulsar Memory if variable is already set

2018-05-22 Thread GitBox 
sijie closed pull request #1831: Don't set Pulsar Memory if variable is already 
set
URL: https://github.com/apache/incubator-pulsar/pull/1831
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/conf/pulsar_tools_env.sh b/conf/pulsar_tools_env.sh
index 7eca6071d2..7ad022ff43 100755
--- a/conf/pulsar_tools_env.sh
+++ b/conf/pulsar_tools_env.sh
@@ -42,7 +42,7 @@
 # PULSAR_GLOBAL_ZK_CONF=
 
 # Extra options to be passed to the jvm
-PULSAR_MEM=" -Xmx256m -XX:MaxDirectMemorySize=256m"
+PULSAR_MEM=${PULSAR_MEM:-"-Xmx256m -XX:MaxDirectMemorySize=256m"}
 
 # Garbage collection options
 PULSAR_GC=" -client "


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1821: Closing client when Proxyconnection is disconnected

2018-05-22 Thread GitBox 
merlimat commented on a change in pull request #1821: Closing client when 
Proxyconnection is disconnected
URL: https://github.com/apache/incubator-pulsar/pull/1821#discussion_r190065269
 
 

 ##
 File path: 
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConnection.java
 ##
 @@ -140,6 +140,10 @@ public void channelInactive(ChannelHandlerContext ctx) 
throws Exception {
 directProxyHandler.outboundChannel.close();
 }
 
+if (client != null) {
+client.close();
+}
 
 Review comment:
   Uhm, that doesn't sound a good option to me


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1827: Added option for custom loggers in C++ and C APIs

2018-05-22 Thread GitBox 
merlimat closed pull request #1827: Added option for custom loggers in C++ and 
C APIs
URL: https://github.com/apache/incubator-pulsar/pull/1827
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pulsar-client-cpp/CMakeLists.txt b/pulsar-client-cpp/CMakeLists.txt
index e5fd716d8e..38a47742ef 100644
--- a/pulsar-client-cpp/CMakeLists.txt
+++ b/pulsar-client-cpp/CMakeLists.txt
@@ -27,6 +27,9 @@ MESSAGE(STATUS "BUILD_TESTS:  " ${BUILD_TESTS})
 option(LINK_STATIC "Link against static libraries" OFF)
 MESSAGE(STATUS "LINK_STATIC:  " ${LINK_STATIC})
 
+option(USE_LOG4CXX "Build with Log4cxx support" OFF)
+MESSAGE(STATUS "USE_LOG4CXX:  " ${USE_LOG4CXX})
+
 IF (CMAKE_BUILD_TYPE STREQUAL "")
 set(CMAKE_BUILD_TYPE RelWithDebInfo)
 ENDIF ()
@@ -65,17 +68,20 @@ if (LINK_STATIC)
 find_library(PROTOBUF_LIBRARIES NAMES libprotobuf.a)
 find_library(CURL_LIBRARY_PATH NAMES libcurl.a curl)
 find_library(LIB_JSON NAMES libjsoncpp.a libjsoncpp_static.a)
-find_library(LOG4CXX_LIBRARY_PATH NAMES liblog4cxx.a)
-
-# Libraries needed by log4cxx to link statically with
-find_library(APR_LIBRARY_PATH NAMES libapr-1.a PATHS /usr/lib 
/usr/local/apr/lib /usr/local/opt/apr/libexec/lib/)
-find_library(APR_UTIL_LIBRARY_PATH NAMES libaprutil-1.a PATHS /usr/lib 
/usr/local/apr/lib /usr/local/opt/apr-util/libexec/lib/)
-find_library(EXPAT_LIBRARY_PATH NAMES libexpat.a expat)
-if (APPLE)
-find_library(ICONV_LIBRARY_PATH NAMES libiconv.a iconv)
-else ()
-set(ICONV_LIBRARY_PATH )
-endif ()
+
+if (USE_LOG4CXX)
+find_library(LOG4CXX_LIBRARY_PATH NAMES liblog4cxx.a)
+
+# Libraries needed by log4cxx to link statically with
+find_library(APR_LIBRARY_PATH NAMES libapr-1.a PATHS /usr/lib 
/usr/local/apr/lib /usr/local/opt/apr/libexec/lib/)
+find_library(APR_UTIL_LIBRARY_PATH NAMES libaprutil-1.a PATHS /usr/lib 
/usr/local/apr/lib /usr/local/opt/apr-util/libexec/lib/)
+find_library(EXPAT_LIBRARY_PATH NAMES libexpat.a expat)
+if (APPLE)
+find_library(ICONV_LIBRARY_PATH NAMES libiconv.a iconv)
+else ()
+set(ICONV_LIBRARY_PATH )
+endif (APPLE)
+endif (USE_LOG4CXX)
 else()
 # Link to shared libraries
 find_package(ZLIB REQUIRED)
@@ -90,7 +96,9 @@ else()
 find_library(LIB_JSON jsoncpp)
 find_library(LOG4CXX_LIBRARY_PATH log4cxx)
 find_library(CURL_LIBRARY_PATH curl)
-find_path(LOG4CXX_INCLUDE_PATH log4cxx/logger.h)
+if (USE_LOG4CXX)
+find_path(LOG4CXX_INCLUDE_PATH log4cxx/logger.h)
+endif (USE_LOG4CXX)
 endif (LINK_STATIC)
 
 
@@ -140,7 +148,11 @@ if (BUILD_TESTS)
 endif ()
 
 find_path(JSON_INCLUDE_PATH jsoncpp)
-find_path(LOG4CXX_INCLUDE_PATH log4cxx/logger.h)
+
+if (USE_LOG4CXX)
+set(CMAKE_CXX_FLAGS " -DUSE_LOG4CXX ${CMAKE_CXX_FLAGS}")
+find_path(LOG4CXX_INCLUDE_PATH log4cxx/logger.h)
+endif (USE_LOG4CXX)
 
 if (NOT LIB_JSON)
 find_library(LIB_JSON json_cpp)
diff --git a/pulsar-client-cpp/include/pulsar/ClientConfiguration.h 
b/pulsar-client-cpp/include/pulsar/ClientConfiguration.h
index 137d528ca2..eedf716240 100644
--- a/pulsar-client-cpp/include/pulsar/ClientConfiguration.h
+++ b/pulsar-client-cpp/include/pulsar/ClientConfiguration.h
@@ -20,6 +20,8 @@
 #define PULSAR_CLIENTCONFIGURATION_H_
 
 #include 
+#include 
+
 #pragma GCC visibility push(default)
 namespace pulsar {
 class PulsarWrapper;
@@ -105,6 +107,7 @@ class ClientConfiguration {
  * Initialize the log configuration
  *
  * @param logConfFilePath  path of the configuration file
+ * @deprecated
  */
 ClientConfiguration& setLogConfFilePath(const std::string& 
logConfFilePath);
 
@@ -113,6 +116,16 @@ class ClientConfiguration {
  */
 const std::string& getLogConfFilePath() const;
 
+/**
+ * Configure a custom logger backend to route of Pulsar client library
+ * to a different logger implementation.
+ *
+ * By default, log messages are printed on standard output.
+ */
+ClientConfiguration& setLogger(LoggerFactoryPtr loggerFactory);
+
+LoggerFactoryPtr getLogger() const;
+
 ClientConfiguration& setUseTls(bool useTls);
 bool isUseTls() const;
 
diff --git a/pulsar-client-cpp/include/pulsar/Logger.h 
b/pulsar-client-cpp/include/pulsar/Logger.h
new file mode 100644
index 00..e319a04f0b
--- /dev/null
+++ b/pulsar-client-cpp/include/pulsar/Logger.h
@@ -0,0 +1,51 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ *

[GitHub] lucperkins opened a new pull request #1832: Update topic names

2018-05-22 Thread GitBox 
lucperkins opened a new pull request #1832: Update topic names
URL: https://github.com/apache/incubator-pulsar/pull/1832
 
 
   This PR updates all example topic names in the docs to use the new system 
(default tenant and namespace, no cluster component, etc.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] srkukarni commented on issue #1831: Don't set Pulsar Memory if variable is already set

2018-05-22 Thread GitBox 
srkukarni commented on issue #1831: Don't set Pulsar Memory if variable is 
already set
URL: https://github.com/apache/incubator-pulsar/pull/1831#issuecomment-391091707
 
 
   @sijie 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] srkukarni opened a new pull request #1831: Don't set Pulsar Memory if variable is already set

2018-05-22 Thread GitBox 
srkukarni opened a new pull request #1831: Don't set Pulsar Memory if variable 
is already set
URL: https://github.com/apache/incubator-pulsar/pull/1831
 
 
   ### Motivation
   This allows users to be able to increase memory for certain admin commands 
by overriding PULSAR_MEM in their command line.
   
   ### Modifications
   
   Describe the modifications you've done.
   
   ### Result
   
   After your change, what will change.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: In S3 offloader, dont use InputStream#available for stream length (#1807)

2018-05-22 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 48e1131  In S3 offloader, dont use InputStream#available for stream 
length (#1807)
48e1131 is described below

commit 48e11316170138da7d8bd511356bfa468f4f0426
Author: Ivan Kelly 
AuthorDate: Tue May 22 18:01:02 2018 +0100

In S3 offloader, dont use InputStream#available for stream length (#1807)

According to the javadoc, available() returns the number bytes that
can be read without blocking. It is _not_ the total length of the
stream. While this currently works for the index as it's backed by a
byte buffer, we shouldn't rely on implicit assumptions like that.

Master issue: #1511
---
 .../pulsar/broker/s3offload/OffloadIndexBlock.java | 26 +++---
 .../broker/s3offload/S3ManagedLedgerOffloader.java |  4 ++--
 .../s3offload/impl/OffloadIndexBlockImpl.java  | 12 +-
 3 files changed, 30 insertions(+), 12 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/OffloadIndexBlock.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/OffloadIndexBlock.java
index 944edca..c7e71c7 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/OffloadIndexBlock.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/OffloadIndexBlock.java
@@ -19,8 +19,9 @@
 package org.apache.pulsar.broker.s3offload;
 
 import java.io.Closeable;
-import java.io.IOException;
+import java.io.FilterInputStream;
 import java.io.InputStream;
+import java.io.IOException;
 import org.apache.bookkeeper.client.api.LedgerMetadata;
 import org.apache.bookkeeper.common.annotation.InterfaceStability.Unstable;
 
@@ -38,7 +39,7 @@ public interface OffloadIndexBlock extends Closeable {
  *   | index_magic_header | index_block_len | index_entry_count |
  *   | data_object_size | segment_metadata_length | segment metadata | 
index entries ... |
  */
-InputStream toStream() throws IOException;
+IndexInputStream toStream() throws IOException;
 
 /**
  * Get the related OffloadIndexEntry that contains the given 
messageEntryId.
@@ -59,9 +60,28 @@ public interface OffloadIndexBlock extends Closeable {
  */
 LedgerMetadata getLedgerMetadata();
 
-/**
+/*
  * Get the total size of the data object.
  */
 long getDataObjectLength();
+
+/**
+ * An input stream which knows the size of the stream upfront.
+ */
+public static class IndexInputStream extends FilterInputStream {
+final long streamSize;
+
+public IndexInputStream(InputStream in, long streamSize) {
+super(in);
+this.streamSize = streamSize;
+}
+
+/**
+ * @return the number of bytes in the stream.
+ */
+public long getStreamSize() {
+return streamSize;
+}
+}
 }
 
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
index ee82532..9cb1486 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/S3ManagedLedgerOffloader.java
@@ -180,10 +180,10 @@ public class S3ManagedLedgerOffloader implements 
LedgerOffloader {
 
 // upload index block
 try (OffloadIndexBlock index = 
indexBuilder.withDataObjectLength(dataObjectLength).build();
- InputStream indexStream = index.toStream()) {
+ OffloadIndexBlock.IndexInputStream indexStream = 
index.toStream()) {
 // write the index block
 ObjectMetadata metadata = new ObjectMetadata();
-metadata.setContentLength(indexStream.available());
+metadata.setContentLength(indexStream.getStreamSize());
 s3client.putObject(new PutObjectRequest(
 bucket,
 indexBlockKey,
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/OffloadIndexBlockImpl.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/OffloadIndexBlockImpl.java
index 638edc4..3c5f337 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/OffloadIndexBlockImpl.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/OffloadIndexBlockImpl.java
@@ -28,6 +28,7 @@ import io.netty.util.Recycler;
 import io.netty.util.Recycler.Handle;
 import java.io.DataInputStream;
 import java.io.IOException;
+import java.io.FilterInputStream;
 import java.io.InputStream;
 import java.util.ArrayList;
 import

[GitHub] ivankelly opened a new pull request #1829: Integration test for s3offload

2018-05-22 Thread GitBox 
ivankelly opened a new pull request #1829: Integration test for s3offload
URL: https://github.com/apache/incubator-pulsar/pull/1829
 
 
   Starts a cluster with s3offload enabled. Write enough entries to roll
   the ledger. Offload. Manually delete the first ledger. Reboot cluster
   to clear caches. Then read back the whole topic.
   
   Master issue: #1511
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1800: Docker testing init script has wrong service url

2018-05-22 Thread GitBox 
merlimat closed pull request #1800: Docker testing init script has wrong 
service url
URL: https://github.com/apache/incubator-pulsar/pull/1800
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/tests/docker-images/latest-version-image/scripts/init-cluster.sh 
b/tests/docker-images/latest-version-image/scripts/init-cluster.sh
index d70e257c8c..24aa0a3959 100755
--- a/tests/docker-images/latest-version-image/scripts/init-cluster.sh
+++ b/tests/docker-images/latest-version-image/scripts/init-cluster.sh
@@ -18,6 +18,8 @@
 # under the License.
 #
 
+set -x
+
 ZNODE="/initialized"
 
 bin/watch-znode.py -z $zkServers -p / -w
@@ -28,7 +30,7 @@ if [ $? != 0 ]; then
 bin/apply-config-from-env.py conf/bookkeeper.conf &&
 bin/pulsar initialize-cluster-metadata --cluster $cluster --zookeeper 
$zkServers \
--configuration-store $configurationStore --web-service-url 
http://$pulsarNode:8080/ \
-   --broker-service-url http://$pulsarNode:6650/ &&
+   --broker-service-url pulsar://$pulsarNode:6650/ &&
 bin/watch-znode.py -z $zkServers -p $ZNODE -c
 echo Initialized
 else


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Docker testing init script has wrong service url (#1800)

2018-05-22 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new e98d50a  Docker testing init script has wrong service url (#1800)
e98d50a is described below

commit e98d50a8cde2fcd0bfc28f2552e35798a1073c81
Author: Ivan Kelly 
AuthorDate: Tue May 22 17:50:26 2018 +0200

Docker testing init script has wrong service url (#1800)

The script was initializing the broker server url with a http url,
using the binary protocol port. This patch fixes that and also logs
the bash commands to make this easier to debug in future.
---
 tests/docker-images/latest-version-image/scripts/init-cluster.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/docker-images/latest-version-image/scripts/init-cluster.sh 
b/tests/docker-images/latest-version-image/scripts/init-cluster.sh
index d70e257..24aa0a3 100755
--- a/tests/docker-images/latest-version-image/scripts/init-cluster.sh
+++ b/tests/docker-images/latest-version-image/scripts/init-cluster.sh
@@ -18,6 +18,8 @@
 # under the License.
 #
 
+set -x
+
 ZNODE="/initialized"
 
 bin/watch-znode.py -z $zkServers -p / -w
@@ -28,7 +30,7 @@ if [ $? != 0 ]; then
 bin/apply-config-from-env.py conf/bookkeeper.conf &&
 bin/pulsar initialize-cluster-metadata --cluster $cluster --zookeeper 
$zkServers \
--configuration-store $configurationStore --web-service-url 
http://$pulsarNode:8080/ \
-   --broker-service-url http://$pulsarNode:6650/ &&
+   --broker-service-url pulsar://$pulsarNode:6650/ &&
 bin/watch-znode.py -z $zkServers -p $ZNODE -c
 echo Initialized
 else

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[GitHub] ivankelly commented on issue #1806: Use our own minimal S3Mock rather than findify S3Mock

2018-05-22 Thread GitBox 
ivankelly commented on issue #1806: Use our own minimal S3Mock rather than 
findify S3Mock
URL: https://github.com/apache/incubator-pulsar/pull/1806#issuecomment-391030648
 
 
   test failing, looks legit. Will look into it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #1799: BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength

2018-05-22 Thread GitBox 
merlimat closed pull request #1799: BlockAwareSegmentInputStreamImpl shouldn't 
use LedgerEntry#getLength
URL: https://github.com/apache/incubator-pulsar/pull/1799
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
index 0fdf48f7f8..03ae702d97 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
@@ -122,14 +122,15 @@ private int readEntries() throws IOException {
 Iterator iterator = ledgerEntriesOnce.iterator();
 while (iterator.hasNext()) {
 LedgerEntry entry = iterator.next();
-int entryLength = (int) entry.getLength();
+ByteBuf buf = entry.getEntryBuffer().retain();
+int entryLength = buf.readableBytes();
 long entryId = entry.getEntryId();
 
 CompositeByteBuf entryBuf = 
PooledByteBufAllocator.DEFAULT.compositeBuffer(2);
 ByteBuf entryHeaderBuf = 
PooledByteBufAllocator.DEFAULT.buffer(ENTRY_HEADER_SIZE, ENTRY_HEADER_SIZE);
 
 entryHeaderBuf.writeInt(entryLength).writeLong(entryId);
-entryBuf.addComponents(true, entryHeaderBuf, 
entry.getEntryBuffer().retain());
+entryBuf.addComponents(true, entryHeaderBuf, buf);
 
 entries.add(entryBuf);
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength (#1799)

2018-05-22 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new ca9099d  BlockAwareSegmentInputStreamImpl shouldn't use 
LedgerEntry#getLength (#1799)
ca9099d is described below

commit ca9099d41586571c9b782a66400d839b08157ea8
Author: Ivan Kelly 
AuthorDate: Tue May 22 15:54:21 2018 +0200

BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength (#1799)

LedgerEntry#getLength() returns the length of the ledger up to and
including the length of the entry, not the length of the entry as you
would expected (and the documentation states). So we shouldn't use it
when serializing the entry, because what we need is the length of the
entry.

This patch changes BlockAwareSegmentInputStreamImpl to use the
readable bytes of the bytebuf instead.

Master Issue: #1511
---
 .../broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java  | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
index 0fdf48f..03ae702 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
@@ -122,14 +122,15 @@ public class BlockAwareSegmentInputStreamImpl extends 
BlockAwareSegmentInputStre
 Iterator iterator = ledgerEntriesOnce.iterator();
 while (iterator.hasNext()) {
 LedgerEntry entry = iterator.next();
-int entryLength = (int) entry.getLength();
+ByteBuf buf = entry.getEntryBuffer().retain();
+int entryLength = buf.readableBytes();
 long entryId = entry.getEntryId();
 
 CompositeByteBuf entryBuf = 
PooledByteBufAllocator.DEFAULT.compositeBuffer(2);
 ByteBuf entryHeaderBuf = 
PooledByteBufAllocator.DEFAULT.buffer(ENTRY_HEADER_SIZE, ENTRY_HEADER_SIZE);
 
 entryHeaderBuf.writeInt(entryLength).writeLong(entryId);
-entryBuf.addComponents(true, entryHeaderBuf, 
entry.getEntryBuffer().retain());
+entryBuf.addComponents(true, entryHeaderBuf, buf);
 
 entries.add(entryBuf);
 }

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.

[GitHub] merlimat commented on a change in pull request #1799: BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength

2018-05-22 Thread GitBox 
merlimat commented on a change in pull request #1799: 
BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength
URL: https://github.com/apache/incubator-pulsar/pull/1799#discussion_r189908975
 
 

 ##
 File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
 ##
 @@ -122,14 +122,15 @@ private int readEntries() throws IOException {
 Iterator iterator = ledgerEntriesOnce.iterator();
 while (iterator.hasNext()) {
 LedgerEntry entry = iterator.next();
-int entryLength = (int) entry.getLength();
+ByteBuf buf = entry.getEntryBuffer().retain();
 
 Review comment:
   Got it, I was missing the `LedgerEntries.close()`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on issue #1800: Docker testing init script has wrong service url

2018-05-22 Thread GitBox 
ivankelly commented on issue #1800: Docker testing init script has wrong 
service url
URL: https://github.com/apache/incubator-pulsar/pull/1800#issuecomment-390998058
 
 
   retest this please // ReplicatorTest.testCloseReplicatorStartProducer


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Use exact dependency version in LICENSE file and use check-binary-license to enforce (#1820)

2018-05-22 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 5ce4065  Use exact dependency version in LICENSE file and use 
check-binary-license to enforce (#1820)
5ce4065 is described below

commit 5ce40658e489c3428887244ed1ec1afaba44c9f9
Author: Matteo Merli 
AuthorDate: Tue May 22 06:37:43 2018 -0700

Use exact dependency version in LICENSE file and use check-binary-license 
to enforce (#1820)
---
 all/src/assemble/LICENSE.bin.txt | 198 +--
 src/check-binary-license |  93 ++
 2 files changed, 244 insertions(+), 47 deletions(-)

diff --git a/all/src/assemble/LICENSE.bin.txt b/all/src/assemble/LICENSE.bin.txt
index dd45fd2..8b0b3cc 100644
--- a/all/src/assemble/LICENSE.bin.txt
+++ b/all/src/assemble/LICENSE.bin.txt
@@ -309,70 +309,174 @@ pulsar-client-cpp/lib/checksum/crc32c_sw.cc
 This projects includes binary packages with the following licenses:
 
 The Apache Software License, Version 2.0
- * JCommander -- com.beust-*.jar
- * High Performance Primitive Collections for Java -- 
com.carrotsearch-hppc-*.jar
- * Jackson -- com.fasterxml.jackson*.jar
- * Caffeine -- com.github.ben-manes.caffeine*.jar
- * Gson -- com.google.code.gson-*.jar
- * Guava -- com.google.guava-*.jar
- * Netty Reactive Streams -- com.typesafe.netty-netty-reactive-streams-*.jar
- * Swagger Annotations -- com.wordnik-swagger-annotations-*.jar
- * Swagger -- io.swagger-swagger-*.jar
- * DataSketches -- com.yahoo.datasketches-*.jar
- * Apache Commons -- commons-*.jar org.apache.commons-*.jar
- * Netty -- io.netty-netty-*.jar
- * Prometheus client -- io.prometheus-simpleclient*.jar
- * Bean Validation API -- javax.validation-*.jar
- * Joda Time -- joda-time-*.jar
- * Log4J -- log4j-*.jar
- * Java Native Access JNA -- net.java.dev.jna-*.jar
- * BookKeeper -- org.apache.bookkeeper.*.jar
- * LZ4 -- net.jpountz.lz4-*.jar
- * AsyncHttpClient -- org.asynchttpclient-*.jar
- * Jetty - org.eclipse.jetty-*.jar
- * SnakeYaml -- org.yaml-snakeyaml-*.jar
- * RocksDB - org.rocksdb.*.jar
- * HttpClient - org.apache.httpcomponents.httpclient.jar
- * HttCore - org.apache.httpcomponents.httpcore.jar
- * CommonsLogging - commons-logging-*.jar
+ * JCommander -- com.beust-jcommander-1.48.jar
+ * High Performance Primitive Collections for Java -- 
com.carrotsearch-hppc-0.7.3.jar
+ * Jackson
+ - com.fasterxml.jackson.core-jackson-annotations-2.8.4.jar
+ - com.fasterxml.jackson.core-jackson-core-2.8.4.jar
+ - com.fasterxml.jackson.core-jackson-databind-2.8.4.jar
+ - com.fasterxml.jackson.dataformat-jackson-dataformat-cbor-2.6.7.jar
+ - com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.8.4.jar
+ - com.fasterxml.jackson.datatype-jackson-datatype-joda-2.8.4.jar
+ - com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.8.4.jar
+ - com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.8.4.jar
+ - com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.8.4.jar
+ - com.fasterxml.jackson.module-jackson-module-jsonSchema-2.9.0.jar
+ * Caffeine -- com.github.ben-manes.caffeine-caffeine-2.3.3.jar
+ * Proto Google Common Protos -- 
com.google.api.grpc-proto-google-common-protos-0.1.9.jar
+ * Gson -- com.google.code.gson-gson-2.8.2.jar
+ * Guava -- com.google.guava-guava-20.0.jar
+ * Netty Reactive Streams -- 
com.typesafe.netty-netty-reactive-streams-2.0.0.jar
+ * Swagger Annotations -- com.wordnik-swagger-annotations-1.5.3-M1.jar
+ * Swagger
+- io.swagger-swagger-annotations-1.5.3.jar
+- io.swagger-swagger-core-1.5.3.jar
+- io.swagger-swagger-models-1.5.3.jar
+ * DataSketches 
+- com.yahoo.datasketches-memory-0.8.3.jar
+- com.yahoo.datasketches-sketches-core-0.8.3.jar
+ * Apache Commons 
+- commons-beanutils-commons-beanutils-1.7.0.jar
+- commons-beanutils-commons-beanutils-core-1.8.0.jar
+- commons-cli-commons-cli-1.2.jar
+- commons-codec-commons-codec-1.10.jar
+- commons-collections-commons-collections-3.2.1.jar
+- commons-configuration-commons-configuration-1.6.jar
+- commons-digester-commons-digester-1.8.jar
+- commons-io-commons-io-2.5.jar
+- commons-lang-commons-lang-2.6.jar
+- commons-logging-commons-logging-1.1.1.jar
+- org.apache.commons-commons-collections4-4.1.jar
+- org.apache.commons-commons-lang3-3.4.jar
+ * Netty
+- io.netty-netty-3.10.1.Final.jar
+- io.netty-netty-all-4.1.21.Final.jar
+- io.netty-netty-codec-http2-4.1.12.Final.jar
+- io.netty-netty-codec-socks-4.1.12.Final.jar
+- io.netty-netty-handler-proxy-4.1.12.Final.jar
+- io.netty-netty-tcnative-boringssl-static-2.0.7.Final.jar
+ * Prometheus client
+- io.prometheus-simpleclient-0.0.23.jar
+- io.prometheus-simpleclient_common-0.0.23.jar
+-

[GitHub] merlimat closed pull request #1820: Use exact dependency version in LICENSE file and use check-binary-license to enforce

2018-05-22 Thread GitBox 
merlimat closed pull request #1820: Use exact dependency version in LICENSE 
file and use check-binary-license to enforce
URL: https://github.com/apache/incubator-pulsar/pull/1820
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/all/src/assemble/LICENSE.bin.txt b/all/src/assemble/LICENSE.bin.txt
index dd45fd2b4e..8b0b3cc5ab 100644
--- a/all/src/assemble/LICENSE.bin.txt
+++ b/all/src/assemble/LICENSE.bin.txt
@@ -309,70 +309,174 @@ pulsar-client-cpp/lib/checksum/crc32c_sw.cc
 This projects includes binary packages with the following licenses:
 
 The Apache Software License, Version 2.0
- * JCommander -- com.beust-*.jar
- * High Performance Primitive Collections for Java -- 
com.carrotsearch-hppc-*.jar
- * Jackson -- com.fasterxml.jackson*.jar
- * Caffeine -- com.github.ben-manes.caffeine*.jar
- * Gson -- com.google.code.gson-*.jar
- * Guava -- com.google.guava-*.jar
- * Netty Reactive Streams -- com.typesafe.netty-netty-reactive-streams-*.jar
- * Swagger Annotations -- com.wordnik-swagger-annotations-*.jar
- * Swagger -- io.swagger-swagger-*.jar
- * DataSketches -- com.yahoo.datasketches-*.jar
- * Apache Commons -- commons-*.jar org.apache.commons-*.jar
- * Netty -- io.netty-netty-*.jar
- * Prometheus client -- io.prometheus-simpleclient*.jar
- * Bean Validation API -- javax.validation-*.jar
- * Joda Time -- joda-time-*.jar
- * Log4J -- log4j-*.jar
- * Java Native Access JNA -- net.java.dev.jna-*.jar
- * BookKeeper -- org.apache.bookkeeper.*.jar
- * LZ4 -- net.jpountz.lz4-*.jar
- * AsyncHttpClient -- org.asynchttpclient-*.jar
- * Jetty - org.eclipse.jetty-*.jar
- * SnakeYaml -- org.yaml-snakeyaml-*.jar
- * RocksDB - org.rocksdb.*.jar
- * HttpClient - org.apache.httpcomponents.httpclient.jar
- * HttCore - org.apache.httpcomponents.httpcore.jar
- * CommonsLogging - commons-logging-*.jar
+ * JCommander -- com.beust-jcommander-1.48.jar
+ * High Performance Primitive Collections for Java -- 
com.carrotsearch-hppc-0.7.3.jar
+ * Jackson
+ - com.fasterxml.jackson.core-jackson-annotations-2.8.4.jar
+ - com.fasterxml.jackson.core-jackson-core-2.8.4.jar
+ - com.fasterxml.jackson.core-jackson-databind-2.8.4.jar
+ - com.fasterxml.jackson.dataformat-jackson-dataformat-cbor-2.6.7.jar
+ - com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.8.4.jar
+ - com.fasterxml.jackson.datatype-jackson-datatype-joda-2.8.4.jar
+ - com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.8.4.jar
+ - com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.8.4.jar
+ - com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.8.4.jar
+ - com.fasterxml.jackson.module-jackson-module-jsonSchema-2.9.0.jar
+ * Caffeine -- com.github.ben-manes.caffeine-caffeine-2.3.3.jar
+ * Proto Google Common Protos -- 
com.google.api.grpc-proto-google-common-protos-0.1.9.jar
+ * Gson -- com.google.code.gson-gson-2.8.2.jar
+ * Guava -- com.google.guava-guava-20.0.jar
+ * Netty Reactive Streams -- 
com.typesafe.netty-netty-reactive-streams-2.0.0.jar
+ * Swagger Annotations -- com.wordnik-swagger-annotations-1.5.3-M1.jar
+ * Swagger
+- io.swagger-swagger-annotations-1.5.3.jar
+- io.swagger-swagger-core-1.5.3.jar
+- io.swagger-swagger-models-1.5.3.jar
+ * DataSketches 
+- com.yahoo.datasketches-memory-0.8.3.jar
+- com.yahoo.datasketches-sketches-core-0.8.3.jar
+ * Apache Commons 
+- commons-beanutils-commons-beanutils-1.7.0.jar
+- commons-beanutils-commons-beanutils-core-1.8.0.jar
+- commons-cli-commons-cli-1.2.jar
+- commons-codec-commons-codec-1.10.jar
+- commons-collections-commons-collections-3.2.1.jar
+- commons-configuration-commons-configuration-1.6.jar
+- commons-digester-commons-digester-1.8.jar
+- commons-io-commons-io-2.5.jar
+- commons-lang-commons-lang-2.6.jar
+- commons-logging-commons-logging-1.1.1.jar
+- org.apache.commons-commons-collections4-4.1.jar
+- org.apache.commons-commons-lang3-3.4.jar
+ * Netty
+- io.netty-netty-3.10.1.Final.jar
+- io.netty-netty-all-4.1.21.Final.jar
+- io.netty-netty-codec-http2-4.1.12.Final.jar
+- io.netty-netty-codec-socks-4.1.12.Final.jar
+- io.netty-netty-handler-proxy-4.1.12.Final.jar
+- io.netty-netty-tcnative-boringssl-static-2.0.7.Final.jar
+ * Prometheus client
+- io.prometheus-simpleclient-0.0.23.jar
+- io.prometheus-simpleclient_common-0.0.23.jar
+- io.prometheus-simpleclient_hotspot-0.0.23.jar
+- io.prometheus-simpleclient_servlet-0.0.23.jar
+ * Bean Validation API -- javax.validation-validation-api-1.1.0.Final.jar
+ * Joda Time -- joda-time-joda-time-2.8.1.jar
+ * Log4J
+- log4j-log4j-1.2.17.jar
+- org.apache.logging.log4j-log4j-api-2.10.0.jar
+- org.apache.logging.log4j-log4j-core-2.10.0.jar
+-

[GitHub] ivankelly commented on a change in pull request #1799: BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength

2018-05-22 Thread GitBox 
ivankelly commented on a change in pull request #1799: 
BlockAwareSegmentInputStreamImpl shouldn't use LedgerEntry#getLength
URL: https://github.com/apache/incubator-pulsar/pull/1799#discussion_r189886758
 
 

 ##
 File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/s3offload/impl/BlockAwareSegmentInputStreamImpl.java
 ##
 @@ -122,14 +122,15 @@ private int readEntries() throws IOException {
 Iterator iterator = ledgerEntriesOnce.iterator();
 while (iterator.hasNext()) {
 LedgerEntry entry = iterator.next();
-int entryLength = (int) entry.getLength();
+ByteBuf buf = entry.getEntryBuffer().retain();
 
 Review comment:
   they are released when LedgerEntries#close gets called. It gets called 
through the try-with-resource block. CompositeByteBuf doesn't retain, so we 
have to call retain explicitly before passing it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch asf-site updated: Updated site at revision 3e04726

2018-05-22 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new c4013e1  Updated site at revision 3e04726
c4013e1 is described below

commit c4013e1cef7282dce4c8a2e15299f00cc0f761f2
Author: jenkins 
AuthorDate: Tue May 22 08:34:06 2018 +

Updated site at revision 3e04726
---
 .../docs/latest/adaptors/PulsarSpark/index.html|  8 
 .../docs/latest/adaptors/PulsarStorm/index.html|  6 +++---
 content/docs/latest/admin-api/overview/index.html  | 10 -
 content/docs/latest/admin/Authz/index.html | 20 +-
 content/docs/latest/clients/Cpp/index.html |  8 
 content/docs/latest/clients/Java/index.html| 12 +--
 content/docs/latest/clients/Python/index.html  |  8 
 content/docs/latest/clients/WebSocket/index.html   |  8 
 .../docs/latest/cookbooks/Encryption/index.html|  6 +++---
 .../latest/cookbooks/PartitionedTopics/index.html  | 14 ++---
 .../latest/cookbooks/RetentionExpiry/index.html| 14 ++---
 .../cookbooks/message-deduplication/index.html | 10 -
 .../docs/latest/cookbooks/message-queue/index.html | 14 ++---
 .../docs/latest/deployment/Kubernetes/index.html   |  4 ++--
 .../docs/latest/deployment/aws-cluster/index.html  |  4 ++--
 content/docs/latest/deployment/cluster/index.html  |  4 ++--
 content/docs/latest/deployment/instance/index.html |  4 ++--
 .../ConceptsAndArchitecture/index.html |  4 ++--
 .../latest/getting-started/LocalCluster/index.html |  4 ++--
 .../latest/getting-started/Pulsar-2.0/index.html   |  2 +-
 .../docs/latest/getting-started/docker/index.html  |  4 ++--
 .../docs/latest/project/BinaryProtocol/index.html  |  4 ++--
 .../docs/latest/project/SimulationTools/index.html |  2 +-
 .../docs/latest/project/schema-storage/index.html  |  4 ++--
 content/docs/latest/reference/CliTools/index.html  | 16 +++
 content/ja/adaptors/PulsarSpark/index.html |  8 
 content/ja/adaptors/PulsarStorm/index.html |  6 +++---
 content/ja/admin/AdminInterface/index.html | 14 ++---
 content/ja/admin/Authz/index.html  | 24 +++---
 content/ja/admin/ClustersBrokers/index.html|  8 
 content/ja/admin/PropertiesNamespaces/index.html   |  8 
 content/ja/advanced/PartitionedTopics/index.html   | 14 ++---
 content/ja/advanced/RetentionExpiry/index.html | 14 ++---
 content/ja/clients/Cpp/index.html  |  8 
 content/ja/clients/Java/index.html |  8 
 content/ja/clients/Python/index.html   |  8 
 content/ja/clients/WebSocket/index.html|  8 
 content/ja/deployment/InstanceSetup/index.html |  8 
 content/ja/deployment/Kubernetes/index.html|  4 ++--
 .../ConceptsAndArchitecture/index.html |  2 +-
 content/ja/getting-started/LocalCluster/index.html |  4 ++--
 content/ja/project/BinaryProtocol/index.html   |  4 ++--
 content/ja/project/SimulationTools/index.html  |  2 +-
 content/ja/reference/CliTools/index.html   | 20 +-
 44 files changed, 183 insertions(+), 183 deletions(-)

diff --git a/content/docs/latest/adaptors/PulsarSpark/index.html 
b/content/docs/latest/adaptors/PulsarSpark/index.html
index 7ce9be9..71c93f7 100644
--- a/content/docs/latest/adaptors/PulsarSpark/index.html
+++ b/content/docs/latest/adaptors/PulsarSpark/index.html
@@ -962,9 +962,9 @@
   
   
   
+  Spark Streaming 
Pulsar receiver
   
   
-  Spark Streaming 
Pulsar receiver
   
   
   
@@ -1184,9 +1184,9 @@
   
   
   
+  Spark Streaming 
Pulsar receiver
   
   
-  Spark Streaming 
Pulsar receiver
   
   
   
@@ -1326,8 +1326,6 @@
   
   
   
-  
-  
   Authentication and 
authorization in Pulsar
   
   
@@ -1480,6 +1478,8 @@
   
   
   
+  
+  
   Using Pulsar as 
a message queue
   
   
diff --git a/content/docs/latest/adaptors/PulsarStorm/index.html 
b/content/docs/latest/adaptors/PulsarStorm/index.html
index e4ac255..b4a40bb 100644
--- a/content/docs/latest/adaptors/PulsarStorm/index.html
+++ b/content/docs/latest/adaptors/PulsarStorm/index.html
@@ -966,9 +966,9 @@
   
   
   
+  Pulsar adaptor for 
Apache Storm
   
   
-  Pulsar adaptor for 
Apache Storm
   
   
   
@@ -1104,8 +1104,6 @@
   
   
   
-  
-

[GitHub] merlimat closed pull request #1819: Support non persistent topics in C++ client

2018-05-22 Thread GitBox 
merlimat closed pull request #1819: Support non persistent topics in C++ client
URL: https://github.com/apache/incubator-pulsar/pull/1819
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pulsar-client-cpp/lib/TopicName.cc 
b/pulsar-client-cpp/lib/TopicName.cc
index 0ad131396b..2258ba80d7 100644
--- a/pulsar-client-cpp/lib/TopicName.cc
+++ b/pulsar-client-cpp/lib/TopicName.cc
@@ -161,7 +161,7 @@ bool TopicName::operator==(const TopicName& other) {
 
 bool TopicName::validate() {
 // check domain matches to "persistent", in future check "memory" when 
server is ready
-if (domain_.compare("persistent") != 0) {
+if (domain_.compare("persistent") != 0 && 
domain_.compare("non-persistent") != 0) {
 return false;
 }
 // cluster_ can be empty
diff --git a/pulsar-client-cpp/tests/BasicEndToEndTest.cc 
b/pulsar-client-cpp/tests/BasicEndToEndTest.cc
index 56c1796835..d4f6a0b058 100644
--- a/pulsar-client-cpp/tests/BasicEndToEndTest.cc
+++ b/pulsar-client-cpp/tests/BasicEndToEndTest.cc
@@ -253,11 +253,11 @@ TEST(BasicEndToEndTest, testNonPersistentTopic) {
 Client client(lookupUrl);
 Producer producer;
 Result result = client.createProducer(topicName, producer);
-ASSERT_EQ(ResultInvalidTopicName, result);
+ASSERT_EQ(ResultOk, result);
 
 Consumer consumer;
 result = client.subscribe(topicName, "my-sub-name", consumer);
-ASSERT_EQ(ResultInvalidTopicName, result);
+ASSERT_EQ(ResultOk, result);
 }
 
 TEST(BasicEndToEndTest, testSingleClientMultipleSubscriptions) {
diff --git a/pulsar-client-cpp/tests/TopicNameTest.cc 
b/pulsar-client-cpp/tests/TopicNameTest.cc
index 5c9410a303..30fcf3a041 100644
--- a/pulsar-client-cpp/tests/TopicNameTest.cc
+++ b/pulsar-client-cpp/tests/TopicNameTest.cc
@@ -78,6 +78,16 @@ TEST(TopicNameTest, testTopicNameV2) {
 ASSERT_EQ(TopicName::getEncodedName("short-topic"), tn1->getLocalName());
 }
 
+TEST(TopicNameTest, testNonPersistentTopicNameV2) {
+// v2 topic names doesn't have "cluster"
+boost::shared_ptr tn1 = 
TopicName::get("non-persistent://tenant/namespace/short-topic");
+ASSERT_EQ("tenant", tn1->getProperty());
+ASSERT_EQ("", tn1->getCluster());
+ASSERT_EQ("namespace", tn1->getNamespacePortion());
+ASSERT_EQ("non-persistent", tn1->getDomain());
+ASSERT_EQ(TopicName::getEncodedName("short-topic"), tn1->getLocalName());
+}
+
 TEST(TopicNameTest, testTopicNameWithSlashes) {
 // Compare getters and setters
 boost::shared_ptr topicName =


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Support non persistent topics in C++ client (#1819)

2018-05-22 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 3e04726  Support non persistent topics in C++ client (#1819)
3e04726 is described below

commit 3e04726c0074839713d4bf6d93fb8953768530b4
Author: Matteo Merli 
AuthorDate: Mon May 21 23:02:32 2018 -0700

Support non persistent topics in C++ client (#1819)

* Support non persistent topics in C++ client

* Fixed end-to-end test with non-persistent topics
---
 pulsar-client-cpp/lib/TopicName.cc   |  2 +-
 pulsar-client-cpp/tests/BasicEndToEndTest.cc |  4 ++--
 pulsar-client-cpp/tests/TopicNameTest.cc | 10 ++
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/pulsar-client-cpp/lib/TopicName.cc 
b/pulsar-client-cpp/lib/TopicName.cc
index 0ad1313..2258ba8 100644
--- a/pulsar-client-cpp/lib/TopicName.cc
+++ b/pulsar-client-cpp/lib/TopicName.cc
@@ -161,7 +161,7 @@ bool TopicName::operator==(const TopicName& other) {
 
 bool TopicName::validate() {
 // check domain matches to "persistent", in future check "memory" when 
server is ready
-if (domain_.compare("persistent") != 0) {
+if (domain_.compare("persistent") != 0 && 
domain_.compare("non-persistent") != 0) {
 return false;
 }
 // cluster_ can be empty
diff --git a/pulsar-client-cpp/tests/BasicEndToEndTest.cc 
b/pulsar-client-cpp/tests/BasicEndToEndTest.cc
index 56c1796..d4f6a0b 100644
--- a/pulsar-client-cpp/tests/BasicEndToEndTest.cc
+++ b/pulsar-client-cpp/tests/BasicEndToEndTest.cc
@@ -253,11 +253,11 @@ TEST(BasicEndToEndTest, testNonPersistentTopic) {
 Client client(lookupUrl);
 Producer producer;
 Result result = client.createProducer(topicName, producer);
-ASSERT_EQ(ResultInvalidTopicName, result);
+ASSERT_EQ(ResultOk, result);
 
 Consumer consumer;
 result = client.subscribe(topicName, "my-sub-name", consumer);
-ASSERT_EQ(ResultInvalidTopicName, result);
+ASSERT_EQ(ResultOk, result);
 }
 
 TEST(BasicEndToEndTest, testSingleClientMultipleSubscriptions) {
diff --git a/pulsar-client-cpp/tests/TopicNameTest.cc 
b/pulsar-client-cpp/tests/TopicNameTest.cc
index 5c9410a..30fcf3a 100644
--- a/pulsar-client-cpp/tests/TopicNameTest.cc
+++ b/pulsar-client-cpp/tests/TopicNameTest.cc
@@ -78,6 +78,16 @@ TEST(TopicNameTest, testTopicNameV2) {
 ASSERT_EQ(TopicName::getEncodedName("short-topic"), tn1->getLocalName());
 }
 
+TEST(TopicNameTest, testNonPersistentTopicNameV2) {
+// v2 topic names doesn't have "cluster"
+boost::shared_ptr tn1 = 
TopicName::get("non-persistent://tenant/namespace/short-topic");
+ASSERT_EQ("tenant", tn1->getProperty());
+ASSERT_EQ("", tn1->getCluster());
+ASSERT_EQ("namespace", tn1->getNamespacePortion());
+ASSERT_EQ("non-persistent", tn1->getDomain());
+ASSERT_EQ(TopicName::getEncodedName("short-topic"), tn1->getLocalName());
+}
+
 TEST(TopicNameTest, testTopicNameWithSlashes) {
 // Compare getters and setters
 boost::shared_ptr topicName =

-- 
To stop receiving notification emails like this one, please contact
mme...@apache.org.