[GitHub] sijie commented on a change in pull request #2347: Fix: function with multi-topic not acking on effectively-once
sijie commented on a change in pull request #2347: Fix: function with
multi-topic not acking on effectively-once
URL: https://github.com/apache/incubator-pulsar/pull/2347#discussion_r208791550
##
File path:
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java
##
@@ -142,8 +140,18 @@ public void open(Map config,
SourceContext sourceContext) throws
.message(message)
.topicName(topicName)
.ackFunction(() -> {
-if (pulsarSourceConfig.getProcessingGuarantees() ==
FunctionConfig.ProcessingGuarantees.EFFECTIVELY_ONCE) {
-inputConsumer.acknowledgeCumulativeAsync(message);
+if (pulsarSourceConfig
+.getProcessingGuarantees() ==
FunctionConfig.ProcessingGuarantees.EFFECTIVELY_ONCE) {
+// try to find actual consumer of the messageId
+if (inputConsumer instanceof MultiTopicsConsumerImpl) {
Review comment:
does this mean ackCumulativeAsync is not working in
MultiTopicsConsumerImpl? If so, should this be fixed in
`MultiTopicsConsumerImpl` instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] rdhabalia opened a new pull request #2347: Fix: function with multi-topic not acking on effectively-once
rdhabalia opened a new pull request #2347: Fix: function with multi-topic not acking on effectively-once URL: https://github.com/apache/incubator-pulsar/pull/2347 ### Motivation `MultiTopicsConsumerImpl` doesn't support `acknowledgeCumulativeAsync` and therefore, function with multi-topic and `EFFECTIVELY_ONCE` processing is not acking message and failing `EFFECTIVELY_ONCE` behavior. ### Modifications Function should ack message for a specific topic consumer if `inputTopicConsumer` is multi-topic consumer. ### Result Function should able to ack messages for multi-topic consumer when processing-guarantee is `EFFECTIVELY_ONCE` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jiazhai opened a new pull request #2346: Issue #2330: change getTopicName in MultiTopicsConsumer
jiazhai opened a new pull request #2346: Issue #2330: change getTopicName in MultiTopicsConsumer URL: https://github.com/apache/incubator-pulsar/pull/2346 ### Motivation `getTopicName` in `TopicMessageIdImpl` and `TopicMessageImpl` is some kind of confusing. Developer may mis-use it. ### Modifications change method `getTopicName` to let it return topic name without partition. add method `getTopicPartitionName`, which return topic name with partition part. ### Result More clear of the method names. Unit-Tests all pass. Issue #2330 get fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: REST and CLI to get function metrics in json for monitoring (#2296)
This is an automated email from the ASF dual-hosted git repository.
rdhabalia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 20b41de REST and CLI to get function metrics in json for monitoring
(#2296)
20b41de is described below
commit 20b41de73912ac1f82280d70a291684cb30d8151
Author: Rajan Dhabalia
AuthorDate: Wed Aug 8 16:57:22 2018 -0700
REST and CLI to get function metrics in json for monitoring (#2296)
* REST and CLI to get function metrics in json for monitoring
* add worker-stats end-point
---
.../apache/pulsar/broker/admin/v2/WorkerStats.java | 47 ++
.../org/apache/pulsar/client/admin/Functions.java | 1 +
.../apache/pulsar/client/admin/PulsarAdmin.java| 11 +++
.../apache/pulsar/client/admin/WorkerStats.java| 35 +++
.../client/admin/internal/FunctionsImpl.java | 2 +-
.../client/admin/internal/WorkerStatsImpl.java | 58
.../pulsar/admin/cli/CmdFunctionWorkerStats.java | 79
.../org/apache/pulsar/admin/cli/CmdFunctions.java | 2 +-
.../apache/pulsar/admin/cli/PulsarAdminTool.java | 1 +
.../src/main/proto/InstanceCommunication.proto | 9 ++
.../pulsar/functions/worker/rest/Resources.java| 2 +
.../functions/worker/rest/api/FunctionsImpl.java | 103 +
.../functions/worker/rest/api/v2/WorkerStats.java | 54 +++
.../org/apache/pulsar/io/kinesis/KinesisSink.java | 4 +-
site2/website/scripts/replace.js | 3 +-
15 files changed, 386 insertions(+), 25 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/WorkerStats.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/WorkerStats.java
new file mode 100644
index 000..962c483
--- /dev/null
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/WorkerStats.java
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.admin.v2;
+
+import java.io.IOException;
+
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.core.Response;
+
+import org.apache.pulsar.functions.proto.InstanceCommunication.Metrics;
+import org.apache.pulsar.functions.worker.rest.FunctionApiResource;
+
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+@Path("/worker-stats")
+public class WorkerStats extends FunctionApiResource {
+
+@GET
+@Path("/functions")
+@ApiOperation(value = "Get metrics for all functions owned by worker",
notes = "Requested should be executed by Monitoring agent on each worker to
fetch the metrics", response = Metrics.class)
+@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have
admin permission"),
+@ApiResponse(code = 503, message = "Worker service is not
running") })
+public Response getMetrics() throws IOException {
+return functions.getFunctionsMetrcis(clientAppId());
+}
+}
diff --git
a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Functions.java
b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Functions.java
index 293304f..c04873d 100644
---
a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Functions.java
+++
b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Functions.java
@@ -27,6 +27,7 @@ import
org.apache.pulsar.client.admin.PulsarAdminException.PreconditionFailedExc
import org.apache.pulsar.common.io.ConnectorDefinition;
import org.apache.pulsar.functions.proto.Function.FunctionDetails;
import
org.apache.pulsar.functions.proto.InstanceCommunication.FunctionStatusList;
+import org.apache.pulsar.functions.proto.InstanceCommunication.Metrics;
import org.apache.pulsar.functions.worker.WorkerInfo;
/**
diff --git
a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java
b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java
index
[GitHub] rdhabalia closed pull request #2296: REST and CLI to get function metrics in json for monitoring
rdhabalia closed pull request #2296: REST and CLI to get function metrics in
json for monitoring
URL: https://github.com/apache/incubator-pulsar/pull/2296
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/WorkerStats.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/WorkerStats.java
new file mode 100644
index 00..962c4835d6
--- /dev/null
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/WorkerStats.java
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.admin.v2;
+
+import java.io.IOException;
+
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.core.Response;
+
+import org.apache.pulsar.functions.proto.InstanceCommunication.Metrics;
+import org.apache.pulsar.functions.worker.rest.FunctionApiResource;
+
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+@Path("/worker-stats")
+public class WorkerStats extends FunctionApiResource {
+
+@GET
+@Path("/functions")
+@ApiOperation(value = "Get metrics for all functions owned by worker",
notes = "Requested should be executed by Monitoring agent on each worker to
fetch the metrics", response = Metrics.class)
+@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have
admin permission"),
+@ApiResponse(code = 503, message = "Worker service is not
running") })
+public Response getMetrics() throws IOException {
+return functions.getFunctionsMetrcis(clientAppId());
+}
+}
diff --git
a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Functions.java
b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Functions.java
index 293304f00f..c04873da8e 100644
---
a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Functions.java
+++
b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Functions.java
@@ -27,6 +27,7 @@
import org.apache.pulsar.common.io.ConnectorDefinition;
import org.apache.pulsar.functions.proto.Function.FunctionDetails;
import
org.apache.pulsar.functions.proto.InstanceCommunication.FunctionStatusList;
+import org.apache.pulsar.functions.proto.InstanceCommunication.Metrics;
import org.apache.pulsar.functions.worker.WorkerInfo;
/**
diff --git
a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java
b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java
index 48aea94289..7a41fe9809 100644
---
a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java
+++
b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java
@@ -43,6 +43,7 @@
import org.apache.pulsar.client.admin.internal.NonPersistentTopicsImpl;
import org.apache.pulsar.client.admin.internal.SchemasImpl;
import org.apache.pulsar.client.admin.internal.TopicsImpl;
+import org.apache.pulsar.client.admin.internal.WorkerStatsImpl;
import org.apache.pulsar.client.admin.internal.TenantsImpl;
import org.apache.pulsar.client.admin.internal.PulsarAdminBuilderImpl;
import org.apache.pulsar.client.admin.internal.ResourceQuotasImpl;
@@ -84,6 +85,7 @@
private final String serviceUrl;
private final Lookup lookups;
private final Functions functions;
+private final WorkerStats workerStats;
private final Schemas schemas;
protected final WebTarget root;
protected final Authentication auth;
@@ -187,6 +189,7 @@ public PulsarAdmin(String serviceUrl,
ClientConfigurationData clientConfigData)
this.resourceQuotas = new ResourceQuotasImpl(root, auth);
this.lookups = new LookupImpl(root, auth, useTls);
this.functions = new FunctionsImpl(root, auth);
+this.workerStats = new WorkerStatsImpl(root, auth);
[GitHub] sijie commented on issue #2283: Improve error message if authorization is not enabled
sijie commented on issue #2283: Improve error message if authorization is not enabled URL: https://github.com/apache/incubator-pulsar/issues/2283#issuecomment-411575999 also need to remove authorizationEnabled since there is already a authProvider setting This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2324: Use pulsar-daemon start zookeeper will use port of 8080
sijie commented on issue #2324: Use pulsar-daemon start zookeeper will use port of 8080 URL: https://github.com/apache/incubator-pulsar/issues/2324#issuecomment-411561820 Close this one since it was answered. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #2345: [documentation] Fix syntax error on tiered storage documentation
sijie closed pull request #2345: [documentation] Fix syntax error on tiered
storage documentation
URL: https://github.com/apache/incubator-pulsar/pull/2345
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/docs/latest/cookbooks/tiered-storage.md
b/site/docs/latest/cookbooks/tiered-storage.md
index 88cb1a3de5..84d2716f7f 100644
--- a/site/docs/latest/cookbooks/tiered-storage.md
+++ b/site/docs/latest/cookbooks/tiered-storage.md
@@ -30,7 +30,7 @@ Offloading is configured in ```broker.conf```.
At a minimum, the administrator must configure the driver, the bucket and the
authenticating credentials. There is also some other knobs to configure, like
the bucket region, the max block size in backed storage, etc.
Currently we support driver of types: { "aws-s3", "google-cloud-storage" },
-{% include admonition.html type="warning" content="Driver names are
case-insensitive for driver's name. There is a third driver type, "s3", which
is identical to "aws-s3", though it requires that you specify an endpoint url
using `s3ManagedLedgerOffloadServiceEndpoint`. This is useful if using a S3
compatible data store, other than AWS." %}
+{% include admonition.html type="warning" content="Driver names are
case-insensitive for driver's name. There is a third driver type, \"s3\", which
is identical to \"aws-s3\", though it requires that you specify an endpoint url
using `s3ManagedLedgerOffloadServiceEndpoint`. This is useful if using a S3
compatible data store, other than AWS." %}
```conf
managedLedgerOffloadDriver=aws-s3
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-pulsar] branch master updated: [documentation] Fix syntax error on tiered storage documentation (#2345)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 3b3f541 [documentation] Fix syntax error on tiered storage
documentation (#2345)
3b3f541 is described below
commit 3b3f541c6dc0d66a3ecaf7a0557fc31e9c8d4dfd
Author: Sijie Guo
AuthorDate: Wed Aug 8 14:13:44 2018 -0700
[documentation] Fix syntax error on tiered storage documentation (#2345)
### Motivation
There is a syntax error in tiered storage doc introduced by #2152.
```
Liquid Exception: Invalid syntax for include tag: type="warning"
content="Driver names are case-insensitive for driver's name. There is a third
driver type, "s3", which is identical to "aws-s3", though it requires that you
specify an endpoint url using `s3ManagedLedgerOffloadServiceEndpoint`. This is
useful if using a S3 compatible data store, other than AWS." Valid syntax: {%
include file.ext param='value' param2='value' %} in
docs/latest/cookbooks/tiered-storage.md
```
### Changes
Fix the syntax error.
---
site/docs/latest/cookbooks/tiered-storage.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/site/docs/latest/cookbooks/tiered-storage.md
b/site/docs/latest/cookbooks/tiered-storage.md
index 88cb1a3..84d2716 100644
--- a/site/docs/latest/cookbooks/tiered-storage.md
+++ b/site/docs/latest/cookbooks/tiered-storage.md
@@ -30,7 +30,7 @@ Offloading is configured in ```broker.conf```.
At a minimum, the administrator must configure the driver, the bucket and the
authenticating credentials. There is also some other knobs to configure, like
the bucket region, the max block size in backed storage, etc.
Currently we support driver of types: { "aws-s3", "google-cloud-storage" },
-{% include admonition.html type="warning" content="Driver names are
case-insensitive for driver's name. There is a third driver type, "s3", which
is identical to "aws-s3", though it requires that you specify an endpoint url
using `s3ManagedLedgerOffloadServiceEndpoint`. This is useful if using a S3
compatible data store, other than AWS." %}
+{% include admonition.html type="warning" content="Driver names are
case-insensitive for driver's name. There is a third driver type, \"s3\", which
is identical to \"aws-s3\", though it requires that you specify an endpoint url
using `s3ManagedLedgerOffloadServiceEndpoint`. This is useful if using a S3
compatible data store, other than AWS." %}
```conf
managedLedgerOffloadDriver=aws-s3
[GitHub] sijie opened a new pull request #2345: [documentation] Fix syntax error on tiered storage documentation
sijie opened a new pull request #2345: [documentation] Fix syntax error on
tiered storage documentation
URL: https://github.com/apache/incubator-pulsar/pull/2345
### Motivation
There is a syntax error in tiered storage doc introduced by #2152.
```
Liquid Exception: Invalid syntax for include tag: type="warning"
content="Driver names are case-insensitive for driver's name. There is a third
driver type, "s3", which is identical to "aws-s3", though it requires that you
specify an endpoint url using `s3ManagedLedgerOffloadServiceEndpoint`. This is
useful if using a S3 compatible data store, other than AWS." Valid syntax: {%
include file.ext param='value' param2='value' %} in
docs/latest/cookbooks/tiered-storage.md
```
### Changes
Fix the syntax error.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] srkukarni commented on issue #2296: REST and CLI to get function metrics in json for monitoring
srkukarni commented on issue #2296: REST and CLI to get function metrics in json for monitoring URL: https://github.com/apache/incubator-pulsar/pull/2296#issuecomment-411502635 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2331: [conf] add zookeeper admin server settings to the configuration files
sijie commented on issue #2331: [conf] add zookeeper admin server settings to the configuration files URL: https://github.com/apache/incubator-pulsar/pull/2331#issuecomment-411501364 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new pull request #2344: [documentation] Cherrypick #2341 to version 2.1 documentation
sijie opened a new pull request #2344: [documentation] Cherrypick #2341 to version 2.1 documentation URL: https://github.com/apache/incubator-pulsar/pull/2344 Cherrypick documentation changes #2341 to 2.1 documentation This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed issue #2252: documentation: incorrect flags for bin/pulsar
sijie closed issue #2252: documentation: incorrect flags for bin/pulsar URL: https://github.com/apache/incubator-pulsar/issues/2252 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: [documentation] Issue 2252: incorrect flags for bin/pulsar (#2341)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 0260df0 [documentation] Issue 2252: incorrect flags for bin/pulsar
(#2341)
0260df0 is described below
commit 0260df0fa8b3fc2a28ac32413be648b535a5c6ba
Author: Sijie Guo
AuthorDate: Wed Aug 8 11:06:29 2018 -0700
[documentation] Issue 2252: incorrect flags for bin/pulsar (#2341)
### Motivation
Fixes #2252
The documentation lists -c and --conf as possible arguments of the
bin/pulsar command.
https://pulsar.incubator.apache.org/docs/v2.0.1-incubating/reference/CliTools/#zookeeper-90gyx
However, the -c and --conf args to bin/pulsar are not respected.
It seems that the proper way to specify an alternative config file is via
env variables,
eg the PULSAR_ZK_CONF env variable.
### Changes
Fix the documentation. Since `bin/pulsar` has specified configuration for
all the components,
we have to use environment variables for overriding the configuration files.
---
site2/docs/reference-cli-tools.md | 61 +--
1 file changed, 26 insertions(+), 35 deletions(-)
diff --git a/site2/docs/reference-cli-tools.md
b/site2/docs/reference-cli-tools.md
index bc3394e..7e3c35b 100644
--- a/site2/docs/reference-cli-tools.md
+++ b/site2/docs/reference-cli-tools.md
@@ -45,7 +45,7 @@ Commands:
Example:
```bash
-$ pulsar broker --conf /path/to/broker.conf
+$ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker
```
The table below lists the environment variables that you can use to configure
the `pulsar` tool.
@@ -80,17 +80,15 @@ Options
|Option|Description|Default|
|---|---|---|
-|`-c`, `--config`|Configuration for the bookie server||
|`-readOnly`|Force start a read-only bookie server|false|
|`-withAutoRecovery`|Start auto-recover service bookie server|false|
Example
```bash
-$ pulsar bookie \
+$ PULSAR_BOOKKEEPER_CONF=/path/to/bookkeeper.conf pulsar bookie \
-readOnly \
- -withAutoRecovery \
- --conf /path/to/bookkeeper.conf
+ -withAutoRecovery
```
### `broker`
@@ -105,14 +103,13 @@ $ pulsar broker options
Options
|Option|Description|Default|
|---|---|---|
-|`-c` , `--broker-conf`|Configuration file for the broker||
|`-bc` , `--bookie-conf`|Configuration file for BookKeeper||
|`-rb` , `--run-bookie`|Run a BookKeeper bookie on the same host as the Pulsar
broker|false|
|`-ra` , `--run-bookie-autorecovery`|Run a BookKeeper autorecovery daemon on
the same host as the Pulsar broker|false|
Example
```bash
-$ pulsar broker --broker-conf /path/to/broker.conf
+$ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker
```
### `compact-topic`
@@ -127,7 +124,6 @@ Options
|Flag|Description|Default|
|---|---|---|
|`-t` , `--topic`|The Pulsar topic that you would like to compact||
-|`-c` , `--broker-conf`|Configuration file for the
broker|${pulsarDirectory}/conf/broker.conf|
Example
```bash
@@ -140,17 +136,12 @@ Run a discovery server
Usage
```bash
-$ pulsar discovery options
+$ pulsar discovery
```
-Options
-|Flag|Description|Default|
-|---|---|---|
-|`-c` , `--conf`|Configuration file for the discovery service||
-
Example
```bash
-$ pulsar discovery --conf /path/to/discovery.conf
+$ PULSAR_DISCOVERY_CONF=/path/to/discovery.conf pulsar discovery
```
### `configuration-store`
@@ -159,14 +150,13 @@ Starts up the Pulsar configuration store
Usage
```bash
-$ pulsar configuration-store options
+$ pulsar configuration-store
```
-Options
-|Flag|Description|Default|
-|---|---|---|
-|`-c` , `--conf`|Configuration file for the configuration store||
-
+Example
+```bash
+$ PULSAR_CONFIGURATION_STORE_CONF=/path/to/configuration_store.conf pulsar
configuration-store
+```
### `initialize-cluster-metadata`
@@ -201,13 +191,12 @@ $ pulsar proxy options
Options
|Flag|Description|Default|
|---|---|---|
-|`-c` , `--config`|Path to a Pulsar proxy configuration file||
|`--configuration-store`|Configuration store connection string||
|`-zk` , `--zookeeper-servers`|Local ZooKeeper connection string||
Example
```bash
-$ pulsar proxy \
+$ PULSAR_PROXY_CONF=/path/to/proxy.conf pulsar proxy \
--zookeeper-servers zk-0,zk-1,zk2 \
--configuration-store zk-0,zk-1,zk-2
```
@@ -227,7 +216,6 @@ Options
|`-a` , `--advertised-address`|The standalone broker advertised address||
|`--bookkeeper-dir`|Local bookies’ base data
directory|data/standalone/bookeeper|
|`--bookkeeper-port`|Local bookies’ base port|3181|
-|`-c` , `--config`|Configuration file path||
|`--no-broker`|Only start ZooKeeper and BookKeeper services, not the
broker|false|
|`--num-bookies`|The number of local bookies|1|
|`--only-broker`|Only start the Pulsar broker service (not ZooKeeper or
BookKeeper)||
@@ -235,19 +223,22 @@ Options
[GitHub] sijie commented on issue #2342: Issue 1067: Problems with Partitioned Topics which name contains -partition-N
sijie commented on issue #2342: Issue 1067: Problems with Partitioned Topics which name contains -partition-N URL: https://github.com/apache/incubator-pulsar/pull/2342#issuecomment-411499082 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new issue #2343: Enable state in cluster mode
sijie opened a new issue #2343: Enable state in cluster mode URL: https://github.com/apache/incubator-pulsar/issues/2343 A couple of things missing on enabling state in cluster mode when I am working on #2335. This is the master ticket for tracking the work items for enabling state in cluster mode. - [ ]: initialize cluster should also initialize the metadata needed for enabling table service in bookkeeper. - [ ]: include tools from bookkeeper for managing state. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni edited a comment on issue #2342: Issue 1067: Problems with Partitioned Topics which name contains -partition-N
srkukarni edited a comment on issue #2342: Issue 1067: Problems with Partitioned Topics which name contains -partition-N URL: https://github.com/apache/incubator-pulsar/pull/2342#issuecomment-411416988 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2342: Issue 1067: Problems with Partitioned Topics which name contains -partition-N
srkukarni commented on issue #2342: Issue 1067: Problems with Partitioned Topics which name contains -partition-N URL: https://github.com/apache/incubator-pulsar/pull/2342#issuecomment-411416988 rerun java tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2341: [documentation] Issue 2252: incorrect flags for bin/pulsar
srkukarni commented on issue #2341: [documentation] Issue 2252: incorrect flags for bin/pulsar URL: https://github.com/apache/incubator-pulsar/pull/2341#issuecomment-411416401 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jiazhai commented on issue #2311: [cpp] receiver queue size config acorss partitions in multi-topics-consumer
jiazhai commented on issue #2311: [cpp] receiver queue size config acorss partitions in multi-topics-consumer URL: https://github.com/apache/incubator-pulsar/pull/2311#issuecomment-411383832 @sijie added python config This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jiazhai commented on a change in pull request #2311: [cpp] receiver queue size config acorss partitions in multi-topics-consumer
jiazhai commented on a change in pull request #2311: [cpp] receiver queue size
config acorss partitions in multi-topics-consumer
URL: https://github.com/apache/incubator-pulsar/pull/2311#discussion_r208520704
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
##
@@ -665,24 +665,29 @@ private boolean topicNameValid(String topicName) {
return subscribeResult;
}
-private void subscribeTopicPartitions(CompletableFuture
subscribeResult, String topicName, int partitionNumber) {
+private void subscribeTopicPartitions(CompletableFuture
subscribeResult, String topicName, int numPartitions) {
if (log.isDebugEnabled()) {
-log.debug("Subscribe to topic {} metadata.partitions: {}",
topicName, partitionNumber);
+log.debug("Subscribe to topic {} metadata.partitions: {}",
topicName, numPartitions);
}
List>> futureList;
-if (partitionNumber > 1) {
-this.topics.putIfAbsent(topicName, partitionNumber);
-allTopicPartitionsNumber.addAndGet(partitionNumber);
+if (numPartitions > 1) {
Review comment:
Thanks. PartitionConsumer is using MultiTopicsConsumer now, and seems the
configuration of queue size across partitions was removed. this is a work to
add it back.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sijie opened a new pull request #2342: Issue 1067: Problems with Partitioned Topics which name contains -partition-N
sijie opened a new pull request #2342: Issue 1067: Problems with Partitioned Topics which name contains -partition-N URL: https://github.com/apache/incubator-pulsar/pull/2342 ### Motivation Fixes #1067. Someone accidentally created a partitioned topic in one of our cluster with a name which contains -partition-2. This raised all sorts of issues, it would seem that only one partition was created for this topic, but metadata exists saying that it has 10 partitions. ### Changes Disallow creating a partitioned topic contains '-partition-' in the name. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new pull request #2341: [documentation] Issue 2252: incorrect flags for bin/pulsar
sijie opened a new pull request #2341: [documentation] Issue 2252: incorrect flags for bin/pulsar URL: https://github.com/apache/incubator-pulsar/pull/2341 ### Motivation Fixes #2252 The documentation lists -c and --conf as possible arguments of the bin/pulsar command. https://pulsar.incubator.apache.org/docs/v2.0.1-incubating/reference/CliTools/#zookeeper-90gyx However, the -c and --conf args to bin/pulsar are not respected. It seems that the proper way to specify an alternative config file is via env variables, eg the PULSAR_ZK_CONF env variable. ### Changes Fix the documentation. Since `bin/pulsar` has specified configuration for all the components, we have to use environment variables for overriding the configuration files. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #2152: GCS offload support(4): add documentations for GCS
sijie closed pull request #2152: GCS offload support(4): add documentations for
GCS
URL: https://github.com/apache/incubator-pulsar/pull/2152
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/docs/latest/cookbooks/tiered-storage.md
b/site/docs/latest/cookbooks/tiered-storage.md
index 4533fa52c2..88cb1a3de5 100644
--- a/site/docs/latest/cookbooks/tiered-storage.md
+++ b/site/docs/latest/cookbooks/tiered-storage.md
@@ -5,6 +5,8 @@ tags: [admin, tiered-storage]
Pulsar's **Tiered Storage** feature allows older backlog data to be offloaded
to long term storage, thereby freeing up space in BookKeeper and reducing
storage costs. This cookbook walks you through using tiered storage in your
Pulsar cluster.
+Tiered storage currently uses [Apache Jclouds](https://jclouds.apache.org) to
supports [Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud
Storage](https://cloud.google.com/storage/)(GCS for short) for long term
storage. With Jclouds, it is easy to add support for more [cloud storage
providers](https://jclouds.apache.org/reference/providers/#blobstore-providers)
in the future.
+
## When should I use Tiered Storage?
Tiered storage should be used when you have a topic for which you want to keep
a very long backlog for a long time. For example, if you have a topic
containing user actions which you use to train your recommendation systems, you
may want to keep that data for a long time, so that if you change your
recommendation algorithm you can rerun it against your full user history.
@@ -17,44 +19,44 @@ A topic in Pulsar is backed by a log, known as a managed
ledger. This log is com
The Tiered Storage offloading mechanism takes advantage of this segment
oriented architecture. When offloading is requested, the segments of the log
are copied, one-by-one, to tiered storage. All segments of the log, apart from
the segment currently being written to can be offloaded.
-## Amazon S3
-
-Tiered storage currently supports S3 for long term storage. On the broker, the
administrator must configure a S3 bucket and the AWS region where the bucket
exists. Offloaded data will be placed into this bucket.
-
-The configured S3 bucket must exist before attempting to offload. If it does
not exist, the offload operation will fail.
+On the broker, the administrator must configure the bucket and credentials for
the cloud storage service. The configured bucket must exist before attempting
to offload. If it does not exist, the offload operation will fail.
-Pulsar users multipart objects to update the segment data. It is possible that
a broker could crash while uploading the data. We recommend you add a lifecycle
rule your S3 bucket to expire incomplete multipart upload after a day or two to
avoid getting charged for incomplete uploads.
+Pulsar uses multi-part objects to upload the segment data. It is possible that
a broker could crash while uploading the data. We recommend you add a life
cycle rule your bucket to expire incomplete multi-part upload after a day or
two to avoid getting charged for incomplete uploads.
-### Configuring the broker
+## Configuring the offload driver
Offloading is configured in ```broker.conf```.
-At a minimum, the user must configure the driver, the region and the bucket.
+At a minimum, the administrator must configure the driver, the bucket and the
authenticating credentials. There is also some other knobs to configure, like
the bucket region, the max block size in backed storage, etc.
+
+Currently we support driver of types: { "aws-s3", "google-cloud-storage" },
+{% include admonition.html type="warning" content="Driver names are
case-insensitive for driver's name. There is a third driver type, "s3", which
is identical to "aws-s3", though it requires that you specify an endpoint url
using `s3ManagedLedgerOffloadServiceEndpoint`. This is useful if using a S3
compatible data store, other than AWS." %}
```conf
-managedLedgerOffloadDriver=S3
-s3ManagedLedgerOffloadRegion=eu-west-3
-s3ManagedLedgerOffloadBucket=pulsar-topic-offload
+managedLedgerOffloadDriver=aws-s3
```
-It is also possible to specify the s3 endpoint directly, using
```s3ManagedLedgerOffloadServiceEndpoint```. This is useful if you are using a
non-AWS storage service which provides an S3 compatible API.
+### "aws-s3" Driver configuration
-{% include admonition.html type="warning" content="If the endpoint is
specified directly, then the region must _not_ be set." %}
+ Bucket and Region
-{% include admonition.html type="warning" content="The broker.conf of all
brokers must have the same configuration for driver, region and bucket for
offload to avoid data becoming unavailable as topics move from one broker to
another." %}
[incubator-pulsar] branch master updated: GCS offload support(4): add documentations for GCS (#2152)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 5e1ee37 GCS offload support(4): add documentations for GCS (#2152)
5e1ee37 is described below
commit 5e1ee372775668c3ff5f59d9beb58f600bb6bb1b
Author: Jia Zhai
AuthorDate: Wed Aug 8 16:33:52 2018 +0800
GCS offload support(4): add documentations for GCS (#2152)
This is the 4th part to support Google Cloud Storage offload.
It aims to add documentations for GCS. And it is based on PR #2151
Master Issue: #2067
---
site/docs/latest/cookbooks/tiered-storage.md | 99 +---
1 file changed, 77 insertions(+), 22 deletions(-)
diff --git a/site/docs/latest/cookbooks/tiered-storage.md
b/site/docs/latest/cookbooks/tiered-storage.md
index 4533fa5..88cb1a3 100644
--- a/site/docs/latest/cookbooks/tiered-storage.md
+++ b/site/docs/latest/cookbooks/tiered-storage.md
@@ -5,6 +5,8 @@ tags: [admin, tiered-storage]
Pulsar's **Tiered Storage** feature allows older backlog data to be offloaded
to long term storage, thereby freeing up space in BookKeeper and reducing
storage costs. This cookbook walks you through using tiered storage in your
Pulsar cluster.
+Tiered storage currently uses [Apache Jclouds](https://jclouds.apache.org) to
supports [Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud
Storage](https://cloud.google.com/storage/)(GCS for short) for long term
storage. With Jclouds, it is easy to add support for more [cloud storage
providers](https://jclouds.apache.org/reference/providers/#blobstore-providers)
in the future.
+
## When should I use Tiered Storage?
Tiered storage should be used when you have a topic for which you want to keep
a very long backlog for a long time. For example, if you have a topic
containing user actions which you use to train your recommendation systems, you
may want to keep that data for a long time, so that if you change your
recommendation algorithm you can rerun it against your full user history.
@@ -17,44 +19,44 @@ A topic in Pulsar is backed by a log, known as a managed
ledger. This log is com
The Tiered Storage offloading mechanism takes advantage of this segment
oriented architecture. When offloading is requested, the segments of the log
are copied, one-by-one, to tiered storage. All segments of the log, apart from
the segment currently being written to can be offloaded.
-## Amazon S3
-
-Tiered storage currently supports S3 for long term storage. On the broker, the
administrator must configure a S3 bucket and the AWS region where the bucket
exists. Offloaded data will be placed into this bucket.
-
-The configured S3 bucket must exist before attempting to offload. If it does
not exist, the offload operation will fail.
+On the broker, the administrator must configure the bucket and credentials for
the cloud storage service. The configured bucket must exist before attempting
to offload. If it does not exist, the offload operation will fail.
-Pulsar users multipart objects to update the segment data. It is possible that
a broker could crash while uploading the data. We recommend you add a lifecycle
rule your S3 bucket to expire incomplete multipart upload after a day or two to
avoid getting charged for incomplete uploads.
+Pulsar uses multi-part objects to upload the segment data. It is possible that
a broker could crash while uploading the data. We recommend you add a life
cycle rule your bucket to expire incomplete multi-part upload after a day or
two to avoid getting charged for incomplete uploads.
-### Configuring the broker
+## Configuring the offload driver
Offloading is configured in ```broker.conf```.
-At a minimum, the user must configure the driver, the region and the bucket.
+At a minimum, the administrator must configure the driver, the bucket and the
authenticating credentials. There is also some other knobs to configure, like
the bucket region, the max block size in backed storage, etc.
+
+Currently we support driver of types: { "aws-s3", "google-cloud-storage" },
+{% include admonition.html type="warning" content="Driver names are
case-insensitive for driver's name. There is a third driver type, "s3", which
is identical to "aws-s3", though it requires that you specify an endpoint url
using `s3ManagedLedgerOffloadServiceEndpoint`. This is useful if using a S3
compatible data store, other than AWS." %}
```conf
-managedLedgerOffloadDriver=S3
-s3ManagedLedgerOffloadRegion=eu-west-3
-s3ManagedLedgerOffloadBucket=pulsar-topic-offload
+managedLedgerOffloadDriver=aws-s3
```
-It is also possible to specify the s3 endpoint directly, using
```s3ManagedLedgerOffloadServiceEndpoint```. This is useful if you are using a
non-AWS storage service which provides an S3 compatible API.
+### "aws-s3" Driver
[incubator-pulsar] branch master updated: Fix flaky test PersistentTopicE2ETest.testMessageRedelivery (#2320)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new d81bf6d Fix flaky test PersistentTopicE2ETest.testMessageRedelivery
(#2320)
d81bf6d is described below
commit d81bf6d9e61edc96e419f078760ae9371e62f566
Author: Matteo Merli
AuthorDate: Wed Aug 8 01:32:21 2018 -0700
Fix flaky test PersistentTopicE2ETest.testMessageRedelivery (#2320)
### Motivation
Test failures are caused by the delayed acks that making the broker
redelivering few more messages than what remained unacked. Disabling delayed
acked to have the behavior expected by the test.
Example of failures:
```
java.lang.AssertionError: expected [my-message-0] but found [my-message-2]
at org.testng.Assert.fail(Assert.java:96)
at org.testng.Assert.failNotEquals(Assert.java:776)
at org.testng.Assert.assertEqualsImpl(Assert.java:137)
at org.testng.Assert.assertEquals(Assert.java:118)
at org.testng.Assert.assertEquals(Assert.java:453)
at org.testng.Assert.assertEquals(Assert.java:463)
at
org.apache.pulsar.broker.service.PersistentTopicE2ETest.testMessageRedelivery(PersistentTopicE2ETest.java:1252)
```
https://builds.apache.org/job/pulsar_precommit_java8/2976/testReport/junit/org.apache.pulsar.broker.service/PersistentTopicE2ETest/testMessageRedelivery_2/
---
.../broker/service/PersistentTopicE2ETest.java | 29 --
1 file changed, 16 insertions(+), 13 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicE2ETest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicE2ETest.java
index a5beefa..ee9fb74 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicE2ETest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicE2ETest.java
@@ -1211,30 +1211,33 @@ public class PersistentTopicE2ETest extends
BrokerTestBase {
* 1. produce messages 2. consume messages and ack all except 1 msg 3.
Verification: should replay only 1 unacked
* message
*/
-@Test()
+@Test
public void testMessageRedelivery() throws Exception {
final String topicName = "persistent://prop/ns-abc/topic2";
final String subName = "sub2";
-Message msg;
+Message msg;
int totalMessages = 10;
-Consumer consumer =
pulsarClient.newConsumer().topic(topicName).subscriptionName(subName)
-.subscriptionType(SubscriptionType.Shared).subscribe();
-Producer producer = pulsarClient.newProducer()
-.topic(topicName)
-.enableBatching(false)
-.messageRoutingMode(MessageRoutingMode.SinglePartition)
-.create();
+Consumer consumer = pulsarClient.newConsumer(Schema.STRING)
+.topic(topicName)
+.subscriptionName(subName)
+.subscriptionType(SubscriptionType.Shared)
+.acknowledgmentGroupTime(0, TimeUnit.SECONDS)
+.subscribe();
+Producer producer = pulsarClient.newProducer(Schema.STRING)
+.topic(topicName)
+.enableBatching(false)
+.messageRoutingMode(MessageRoutingMode.SinglePartition)
+.create();
// (1) Produce messages
for (int i = 0; i < totalMessages; i++) {
-String message = "my-message-" + i;
-producer.send(message.getBytes());
+producer.send("my-message-" + i);
}
// (2) Consume and ack messages except first message
-Message unAckedMsg = null;
+Message unAckedMsg = null;
for (int i = 0; i < totalMessages; i++) {
msg = consumer.receive();
if (i == 0) {
@@ -1249,7 +1252,7 @@ public class PersistentTopicE2ETest extends
BrokerTestBase {
// Verify: msg [L:0] must be redelivered
try {
msg = consumer.receive(1, TimeUnit.SECONDS);
-assertEquals(new String(msg.getData()), new
String(unAckedMsg.getData()));
+assertEquals(msg.getValue(), unAckedMsg.getValue());
} catch (Exception e) {
fail("msg should be redelivered ", e);
}
[GitHub] sijie closed pull request #2320: Fix flaky test PersistentTopicE2ETest.testMessageRedelivery
sijie closed pull request #2320: Fix flaky test
PersistentTopicE2ETest.testMessageRedelivery
URL: https://github.com/apache/incubator-pulsar/pull/2320
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicE2ETest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicE2ETest.java
index a5beefaceb..ee9fb74c5c 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicE2ETest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicE2ETest.java
@@ -1211,30 +1211,33 @@ public void testPayloadCorruptionDetection() throws
Exception {
* 1. produce messages 2. consume messages and ack all except 1 msg 3.
Verification: should replay only 1 unacked
* message
*/
-@Test()
+@Test
public void testMessageRedelivery() throws Exception {
final String topicName = "persistent://prop/ns-abc/topic2";
final String subName = "sub2";
-Message msg;
+Message msg;
int totalMessages = 10;
-Consumer consumer =
pulsarClient.newConsumer().topic(topicName).subscriptionName(subName)
-.subscriptionType(SubscriptionType.Shared).subscribe();
-Producer producer = pulsarClient.newProducer()
-.topic(topicName)
-.enableBatching(false)
-.messageRoutingMode(MessageRoutingMode.SinglePartition)
-.create();
+Consumer consumer = pulsarClient.newConsumer(Schema.STRING)
+.topic(topicName)
+.subscriptionName(subName)
+.subscriptionType(SubscriptionType.Shared)
+.acknowledgmentGroupTime(0, TimeUnit.SECONDS)
+.subscribe();
+Producer producer = pulsarClient.newProducer(Schema.STRING)
+.topic(topicName)
+.enableBatching(false)
+.messageRoutingMode(MessageRoutingMode.SinglePartition)
+.create();
// (1) Produce messages
for (int i = 0; i < totalMessages; i++) {
-String message = "my-message-" + i;
-producer.send(message.getBytes());
+producer.send("my-message-" + i);
}
// (2) Consume and ack messages except first message
-Message unAckedMsg = null;
+Message unAckedMsg = null;
for (int i = 0; i < totalMessages; i++) {
msg = consumer.receive();
if (i == 0) {
@@ -1249,7 +1252,7 @@ public void testMessageRedelivery() throws Exception {
// Verify: msg [L:0] must be redelivered
try {
msg = consumer.receive(1, TimeUnit.SECONDS);
-assertEquals(new String(msg.getData()), new
String(unAckedMsg.getData()));
+assertEquals(msg.getValue(), unAckedMsg.getValue());
} catch (Exception e) {
fail("msg should be redelivered ", e);
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sijie commented on issue #2331: [conf] add zookeeper admin server settings to the configuration files
sijie commented on issue #2331: [conf] add zookeeper admin server settings to the configuration files URL: https://github.com/apache/incubator-pulsar/pull/2331#issuecomment-411329730 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Introduce warmup period in performance producer (#2338)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new acc26c9 Introduce warmup period in performance producer (#2338)
acc26c9 is described below
commit acc26c9e08476c87d30045a705c4d11d6a572b69
Author: Matteo Merli
AuthorDate: Wed Aug 8 01:31:11 2018 -0700
Introduce warmup period in performance producer (#2338)
---
.../pulsar/testclient/PerformanceProducer.java | 20 +---
1 file changed, 13 insertions(+), 7 deletions(-)
diff --git
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
index 0643fe7..bfd4fbb 100644
---
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
+++
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
@@ -136,6 +136,9 @@ public class PerformanceProducer {
"--test-duration" }, description = "Test duration in secs. If
0, it will keep publishing")
public long testTime = 0;
+@Parameter(names = "--warmup-time", description = "Warm-up time in
seconds (Default: 1 sec)")
+public double warmupTimeSeconds = 1.0;
+
@Parameter(names = {
"--use-tls" }, description = "Use TLS encryption on the
connection")
public boolean useTls;
@@ -212,8 +215,6 @@ public class PerformanceProducer {
}
}
-arguments.testTime = TimeUnit.SECONDS.toMillis(arguments.testTime);
-
// Dump config variables
ObjectMapper m = new ObjectMapper();
ObjectWriter w = m.writerWithDefaultPrettyPrinter();
@@ -319,14 +320,16 @@ public class PerformanceProducer {
try {
RateLimiter rateLimiter =
RateLimiter.create(arguments.msgRate);
-long startTime = System.currentTimeMillis();
+long startTime = System.nanoTime();
+long warmupEndTime = startTime + (long)
(arguments.warmupTimeSeconds * 1e9);
+long testEndTime = startTime + (long) (arguments.testTime *
1e9);
// Send messages on all topics/producers
long totalSent = 0;
while (true) {
for (Producer producer : producers) {
if (arguments.testTime > 0) {
-if (System.currentTimeMillis() - startTime >
arguments.testTime) {
+if (System.nanoTime() > testEndTime) {
log.info("--- DONE
---");
printAggregatedStats();
isDone.set(true);
@@ -352,9 +355,12 @@ public class PerformanceProducer {
messagesSent.increment();
bytesSent.add(payloadData.length);
-long latencyMicros =
NANOSECONDS.toMicros(System.nanoTime() - sendTime);
-recorder.recordValue(latencyMicros);
-cumulativeRecorder.recordValue(latencyMicros);
+long now = System.nanoTime();
+if (now > warmupEndTime) {
+long latencyMicros = NANOSECONDS.toMicros(now
- sendTime);
+recorder.recordValue(latencyMicros);
+cumulativeRecorder.recordValue(latencyMicros);
+}
}).exceptionally(ex -> {
log.warn("Write error on message", ex);
System.exit(-1);
[GitHub] sijie closed pull request #2337: [website] fix broken links
sijie closed pull request #2337: [website] fix broken links URL: https://github.com/apache/incubator-pulsar/pull/2337 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/site2/website/blog/2018-08-06-Apache-Pulsar-2-1-0.md b/site2/website/blog/2018-08-06-Apache-Pulsar-2-1-0.md index 88ca34a5d0..2fd8444635 100644 --- a/site2/website/blog/2018-08-06-Apache-Pulsar-2-1-0.md +++ b/site2/website/blog/2018-08-06-Apache-Pulsar-2-1-0.md @@ -11,7 +11,7 @@ brought multiple new features and improvements to Pulsar. In Pulsar 2.1 you'll see: - [Pulsar IO](/docs/io-overview) connector framework and a list of [builtin connectors](/docs/io-connectors) -- [PIP-17](https://github.com/apache/incubator-pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics): [Tiered Storage](/docs/docs/concepts-tiered-storage) +- [PIP-17](https://github.com/apache/incubator-pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics): [Tiered Storage](/docs/concepts-tiered-storage) - Pulsar [Stateful Functions](/docs/functions-state) - [Go Client](/docs/client-libraries-go) - [Avro](https://github.com/apache/incubator-pulsar/blob/v2.1.0-incubating/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/AvroSchema.java) @@ -63,7 +63,7 @@ offloading mechanisms and metadata management are transparent to applications. Currently [S3](https://aws.amazon.com/s3/) is supported in 2.1. More offloaders (such as Google GCS, Azure Blobstore, and HDFS) are coming in future releases. -If you are interested in this feature, you can checkout more details [here](cookbooks-tiered-storage). +If you are interested in this feature, you can checkout more details [here](/docs/cookbooks-tiered-storage). ## Stateful Function This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #2338: Introduce warmup period in performance producer
sijie closed pull request #2338: Introduce warmup period in performance producer
URL: https://github.com/apache/incubator-pulsar/pull/2338
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
index 0643fe754b..bfd4ff 100644
---
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
+++
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
@@ -136,6 +136,9 @@
"--test-duration" }, description = "Test duration in secs. If
0, it will keep publishing")
public long testTime = 0;
+@Parameter(names = "--warmup-time", description = "Warm-up time in
seconds (Default: 1 sec)")
+public double warmupTimeSeconds = 1.0;
+
@Parameter(names = {
"--use-tls" }, description = "Use TLS encryption on the
connection")
public boolean useTls;
@@ -212,8 +215,6 @@ public static void main(String[] args) throws Exception {
}
}
-arguments.testTime = TimeUnit.SECONDS.toMillis(arguments.testTime);
-
// Dump config variables
ObjectMapper m = new ObjectMapper();
ObjectWriter w = m.writerWithDefaultPrettyPrinter();
@@ -319,14 +320,16 @@ public void run() {
try {
RateLimiter rateLimiter =
RateLimiter.create(arguments.msgRate);
-long startTime = System.currentTimeMillis();
+long startTime = System.nanoTime();
+long warmupEndTime = startTime + (long)
(arguments.warmupTimeSeconds * 1e9);
+long testEndTime = startTime + (long) (arguments.testTime *
1e9);
// Send messages on all topics/producers
long totalSent = 0;
while (true) {
for (Producer producer : producers) {
if (arguments.testTime > 0) {
-if (System.currentTimeMillis() - startTime >
arguments.testTime) {
+if (System.nanoTime() > testEndTime) {
log.info("--- DONE
---");
printAggregatedStats();
isDone.set(true);
@@ -352,9 +355,12 @@ public void run() {
messagesSent.increment();
bytesSent.add(payloadData.length);
-long latencyMicros =
NANOSECONDS.toMicros(System.nanoTime() - sendTime);
-recorder.recordValue(latencyMicros);
-cumulativeRecorder.recordValue(latencyMicros);
+long now = System.nanoTime();
+if (now > warmupEndTime) {
+long latencyMicros = NANOSECONDS.toMicros(now
- sendTime);
+recorder.recordValue(latencyMicros);
+cumulativeRecorder.recordValue(latencyMicros);
+}
}).exceptionally(ex -> {
log.warn("Write error on message", ex);
System.exit(-1);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-pulsar] branch master updated: [website] fix broken links (#2337)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 8cc7498 [website] fix broken links (#2337) 8cc7498 is described below commit 8cc74988e9d6919a7387832dde9d14602b36c1e8 Author: Sijie Guo AuthorDate: Wed Aug 8 01:30:56 2018 -0700 [website] fix broken links (#2337) --- site2/website/blog/2018-08-06-Apache-Pulsar-2-1-0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site2/website/blog/2018-08-06-Apache-Pulsar-2-1-0.md b/site2/website/blog/2018-08-06-Apache-Pulsar-2-1-0.md index 88ca34a..2fd8444 100644 --- a/site2/website/blog/2018-08-06-Apache-Pulsar-2-1-0.md +++ b/site2/website/blog/2018-08-06-Apache-Pulsar-2-1-0.md @@ -11,7 +11,7 @@ brought multiple new features and improvements to Pulsar. In Pulsar 2.1 you'll see: - [Pulsar IO](/docs/io-overview) connector framework and a list of [builtin connectors](/docs/io-connectors) -- [PIP-17](https://github.com/apache/incubator-pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics): [Tiered Storage](/docs/docs/concepts-tiered-storage) +- [PIP-17](https://github.com/apache/incubator-pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics): [Tiered Storage](/docs/concepts-tiered-storage) - Pulsar [Stateful Functions](/docs/functions-state) - [Go Client](/docs/client-libraries-go) - [Avro](https://github.com/apache/incubator-pulsar/blob/v2.1.0-incubating/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/AvroSchema.java) @@ -63,7 +63,7 @@ offloading mechanisms and metadata management are transparent to applications. Currently [S3](https://aws.amazon.com/s3/) is supported in 2.1. More offloaders (such as Google GCS, Azure Blobstore, and HDFS) are coming in future releases. -If you are interested in this feature, you can checkout more details [here](cookbooks-tiered-storage). +If you are interested in this feature, you can checkout more details [here](/docs/cookbooks-tiered-storage). ## Stateful Function
[incubator-pulsar] branch master updated: added FRench to website (#2339)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 20e652b added FRench to website (#2339)
20e652b is described below
commit 20e652bd7e4b02bb522f3f48df2afcfd695b57d8
Author: danielccyr
AuthorDate: Wed Aug 8 04:30:14 2018 -0400
added FRench to website (#2339)
### Motivation
adding french to website
### Modifications
toggled a flag
---
site2/website/languages.js | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/site2/website/languages.js b/site2/website/languages.js
index 90801ef..8629453 100644
--- a/site2/website/languages.js
+++ b/site2/website/languages.js
@@ -67,7 +67,7 @@ const languages = [
tag: 'fi',
},
{
-enabled: false,
+enabled: true,
name: 'Français',
tag: 'fr',
},
[GitHub] sijie commented on issue #2339: added FRench to website
sijie commented on issue #2339: added FRench to website URL: https://github.com/apache/incubator-pulsar/pull/2339#issuecomment-411329483 @danielccyr thank you for your contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #2339: added FRench to website
sijie closed pull request #2339: added FRench to website
URL: https://github.com/apache/incubator-pulsar/pull/2339
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site2/website/languages.js b/site2/website/languages.js
index 90801eff03..8629453f2c 100644
--- a/site2/website/languages.js
+++ b/site2/website/languages.js
@@ -67,7 +67,7 @@ const languages = [
tag: 'fi',
},
{
-enabled: false,
+enabled: true,
name: 'Français',
tag: 'fr',
},
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sijie commented on issue #2330: Function localrun quits when using partitioned topic.
sijie commented on issue #2330: Function localrun quits when using partitioned topic. URL: https://github.com/apache/incubator-pulsar/issues/2330#issuecomment-411305563 I had a discussion with @codelipenghui . the original question was around stateful function. stateful function was shipped as a preview feature in 2.1. however it is not fully integrated in cluster mode. so I will be working on that piece in #2335 . However there is a bigger problem around `partitioned-topic`. It seems there is some regression in partitioned topic in 2.1, which pulsar functions can't run with `partitioned-topic`. because we switched to using multi-topic subscription in 2.1, so it will return a partition name instead of topic name, which cause function runtime confused on finding a suitable serde to deserialize the message and it throws RuntimeException and quits. I marked this issue as `bug` and updated the issue title to be more specific. this fix should also be cherry-picked to 2.1.1 release as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codelipenghui edited a comment on issue #2330: Functions localrun quit by can't get the reason
codelipenghui edited a comment on issue #2330: Functions localrun quit by can't get the reason URL: https://github.com/apache/incubator-pulsar/issues/2330#issuecomment-411298133 Topic of plat.correctness.verification is a partitioned topic with six partitions. When i set input to plat.correctness.verification local run will quit. When i set input to plat.correctness.verification-partition-0 local run can process success. - Local run success with command below ```shell bin/pulsar-admin functions localrun --jar target/function-starter-1.0.0-all.jar --className com.codelipenghui.pulsar.function.WordCountFunction --inputs persistent://public/default/plat.correctness.verification-partition-0 --output persistent://public/default/plat.correctness.verification.wordcount.output --name plat-correctness-verification-wordcount ``` - Local run quit with command below ```shell bin/pulsar-admin functions localrun --jar target/function-starter-1.0.0-all.jar --className com.codelipenghui.pulsar.function.WordCountFunction --inputs persistent://public/default/plat.correctness.verification --output persistent://public/default/plat.correctness.verification.wordcount.output --name plat-correctness-verification-wordcount ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codelipenghui edited a comment on issue #2330: Functions localrun quit by can't get the reason
codelipenghui edited a comment on issue #2330: Functions localrun quit by can't get the reason URL: https://github.com/apache/incubator-pulsar/issues/2330#issuecomment-411298133 Topic of plat.correctness.verification is a partitioned topic with six partitions. When i set input to plat.correctness.verification local run will quit. When i set input to plat.correctness.verification-partition-0 local run can process success. ```shell # Local run success with command below bin/pulsar-admin functions localrun --jar target/function-starter-1.0.0-all.jar --className com.codelipenghui.pulsar.function.WordCountFunction --inputs persistent://public/default/plat.correctness.verification-partition-0 --output persistent://public/default/plat.correctness.verification.wordcount.output --name plat-correctness-verification-wordcount ``` # Local run quit with command below ```shell bin/pulsar-admin functions localrun --jar target/function-starter-1.0.0-all.jar --className com.codelipenghui.pulsar.function.WordCountFunction --inputs persistent://public/default/plat.correctness.verification --output persistent://public/default/plat.correctness.verification.wordcount.output --name plat-correctness-verification-wordcount ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codelipenghui commented on issue #2330: Functions localrun quit by can't get the reason
codelipenghui commented on issue #2330: Functions localrun quit by can't get the reason URL: https://github.com/apache/incubator-pulsar/issues/2330#issuecomment-411298133 Topic of plat.correctness.verification is a partitioned topic with six partitions. When i set input to plat.correctness.verification local run will quit. When i set input to plat.correctness.verification-partition-0 local run can process success. ```shell # Local run success bin/pulsar-admin functions localrun --jar target/function-starter-1.0.0-all.jar --className com.codelipenghui.pulsar.function.WordCountFunction --inputs persistent://public/default/plat.correctness.verification-partition-0 --output persistent://public/default/plat.correctness.verification.wordcount.output --name plat-correctness-verification-wordcount ``` ```shell # Local run quit bin/pulsar-admin functions localrun --jar target/function-starter-1.0.0-all.jar --className com.codelipenghui.pulsar.function.WordCountFunction --inputs persistent://public/default/plat.correctness.verification --output persistent://public/default/plat.correctness.verification.wordcount.output --name plat-correctness-verification-wordcount ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
