[GitHub] srkukarni commented on issue #2502: Fix Handling of user defined nar sources/sinks
srkukarni commented on issue #2502: Fix Handling of user defined nar sources/sinks URL: https://github.com/apache/incubator-pulsar/pull/2502#issuecomment-418607901 @sijie added an integration test This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2502: Fix Handling of user defined nar sources/sinks
srkukarni commented on issue #2502: Fix Handling of user defined nar sources/sinks URL: https://github.com/apache/incubator-pulsar/pull/2502#issuecomment-418607844 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on issue #2518: Add CLI to get function status of a specific instance
rdhabalia commented on issue #2518: Add CLI to get function status of a specific instance URL: https://github.com/apache/incubator-pulsar/pull/2518#issuecomment-418601341 @srkukarni updated the change. > What about server side changes? Function already has [server side api](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/v2/FunctionApiV2Resource.java#L102). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on issue #2517: Log and Return error-message on function-stats failure
rdhabalia commented on issue #2517: Log and Return error-message on function-stats failure URL: https://github.com/apache/incubator-pulsar/pull/2517#issuecomment-418600423 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia removed a comment on issue #2517: Log and Return error-message on function-stats failure
rdhabalia removed a comment on issue #2517: Log and Return error-message on function-stats failure URL: https://github.com/apache/incubator-pulsar/pull/2517#issuecomment-418600258 rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia removed a comment on issue #2517: Log and Return error-message on function-stats failure
rdhabalia removed a comment on issue #2517: Log and Return error-message on function-stats failure URL: https://github.com/apache/incubator-pulsar/pull/2517#issuecomment-418600423 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on issue #2517: Log and Return error-message on function-stats failure
rdhabalia commented on issue #2517: Log and Return error-message on function-stats failure URL: https://github.com/apache/incubator-pulsar/pull/2517#issuecomment-418600258 rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia removed a comment on issue #2517: Log and Return error-message on function-stats failure
rdhabalia removed a comment on issue #2517: Log and Return error-message on function-stats failure URL: https://github.com/apache/incubator-pulsar/pull/2517#issuecomment-418600193 rerun tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on issue #2517: Log and Return error-message on function-stats failure
rdhabalia commented on issue #2517: Log and Return error-message on function-stats failure URL: https://github.com/apache/incubator-pulsar/pull/2517#issuecomment-418600193 rerun tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on a change in pull request #2518: Add CLI to get function status of a specific instance
srkukarni commented on a change in pull request #2518: Add CLI to get function
status of a specific instance
URL: https://github.com/apache/incubator-pulsar/pull/2518#discussion_r215128833
##
File path:
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
##
@@ -872,9 +872,16 @@ void runCmd() throws Exception {
@Parameters(commandDescription = "Check the current status of a Pulsar
Function")
class GetFunctionStatus extends FunctionCommand {
+
+@Parameter(names = "--instance-id", description = "The function
instanceId (Get-status of all instances if instance-id is not provided")
+protected String instanceId;
+
@Override
void runCmd() throws Exception {
-String json =
Utils.printJson(admin.functions().getFunctionStatus(tenant, namespace,
functionName));
+String json = Utils.printJson(
+isNotBlank(instanceId) ?
admin.functions().getFunctionStatus(tenant, namespace, functionName)
Review comment:
shouldn't it be the other way around
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] srkukarni commented on issue #2514: Use SigKill to kill python process as opposed to sigterm which can be caught
srkukarni commented on issue #2514: Use SigKill to kill python process as opposed to sigterm which can be caught URL: https://github.com/apache/incubator-pulsar/pull/2514#issuecomment-418590116 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia opened a new pull request #2519: Delete temp file after submitting function
rdhabalia opened a new pull request #2519: Delete temp file after submitting function URL: https://github.com/apache/incubator-pulsar/pull/2519 ### Motivation Right now, function-cli doesn't cleanup function-pkg after downloading from http-url. ### Modifications Delete temp file after submitting function. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia opened a new pull request #2518: Add CLI to get function status of a specific instance
rdhabalia opened a new pull request #2518: Add CLI to get function status of a specific instance URL: https://github.com/apache/incubator-pulsar/pull/2518 ### Motivation Add CLI support to get status of specific function instance. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia opened a new pull request #2517: Log and Return error-message on function-stats failure
rdhabalia opened a new pull request #2517: Log and Return error-message on function-stats failure URL: https://github.com/apache/incubator-pulsar/pull/2517 ### Motivation Right now, for some-reason when get-function-status api call fails then , app-log doesn't log exception with function reference and it doesn't give correct failure-response code. ### Modifications Fix error-logging and response code on function get status ap call. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia opened a new pull request #2516: Fix: get function status with auth enable
rdhabalia opened a new pull request #2516: Fix: get function status with auth enable URL: https://github.com/apache/incubator-pulsar/pull/2516 ### Motivation Right now, function getstatus doesn't work with auth enable. ### Result get-function status cli will work with enabled auth. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng opened a new pull request #2515: support nested fields in Pulsar presto connector
jerrypeng opened a new pull request #2515: support nested fields in Pulsar presto connector URL: https://github.com/apache/incubator-pulsar/pull/2515 ### Motivation Support nested fields in POJOs to be able to be queried by sql This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2514: Use SigKill to kill python process as opposed to sigterm which can be caught
srkukarni commented on issue #2514: Use SigKill to kill python process as opposed to sigterm which can be caught URL: https://github.com/apache/incubator-pulsar/pull/2514#issuecomment-418564929 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #2512: Misc Cleanups
srkukarni commented on issue #2512: Misc Cleanups URL: https://github.com/apache/incubator-pulsar/pull/2512#issuecomment-418561459 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #2473: Downgrading ZK to stable version 3.4.13
merlimat commented on issue #2473: Downgrading ZK to stable version 3.4.13 URL: https://github.com/apache/incubator-pulsar/pull/2473#issuecomment-418557602 @ivankelly @sijie Added integration test with ZK 3.5 snapshot and log This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni opened a new pull request #2514: Use SigKill to kill python process as opposed to sigterm which can be caught
srkukarni opened a new pull request #2514: Use SigKill to kill python process as opposed to sigterm which can be caught URL: https://github.com/apache/incubator-pulsar/pull/2514 ### Motivation Explain here the context, and why you're making that change. What is the problem you're trying to solve. ### Modifications Describe the modifications you've done. ### Result After your change, what will change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: [schema] implement generic schema/record for Schema.JSON (#2497)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 8453d73 [schema] implement generic schema/record for Schema.JSON
(#2497)
8453d73 is described below
commit 8453d73e94f0a22de53a4a003c2084e1573ac855
Author: Sijie Guo
AuthorDate: Tue Sep 4 15:56:07 2018 -0700
[schema] implement generic schema/record for Schema.JSON (#2497)
### Motivation
The `GenericSchema` and `GenericRecord` are used for schema auto detection.
Currently it only supports Avro.
This PR is to support json.
### Changes
Introduce generic schema and generic record for json schema
---
.../pulsar/client/api/schema/GenericRecord.java| 4 +-
.../schema/{ => generic}/GenericAvroRecord.java| 7 +-
.../schema/{ => generic}/GenericAvroSchema.java| 33 ++--
.../impl/schema/generic/GenericJsonRecord.java | 74
.../impl/schema/generic/GenericJsonSchema.java | 63 ++
.../client/impl/schema/generic/GenericSchema.java | 77 +
.../impl/schema/generic/GenericSchemaTest.java | 99 ++
.../pulsar/client/schema/AvroSchemaTest.java | 93 +---
.../pulsar/client/schema/JSONSchemaTest.java | 38 ++---
.../pulsar/client/schema/SchemaTestUtils.java | 61 +
.../pulsar/client/impl/PulsarClientImpl.java | 10 +--
11 files changed, 398 insertions(+), 161 deletions(-)
diff --git
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
index 0a4fce4..46a49a1 100644
---
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
+++
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
@@ -38,7 +38,9 @@ public interface GenericRecord {
* @param field the field to retrieve the value
* @return the value object
*/
-Object getField(Field field);
+default Object getField(Field field) {
+return getField(field.getName());
+}
/**
* Retrieve the value of the provided fieldName.
diff --git
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroRecord.java
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
similarity index 94%
rename from
pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroRecord.java
rename to
pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
index fb65c7a..c9dbeb7 100644
---
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroRecord.java
+++
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pulsar.client.impl.schema;
+package org.apache.pulsar.client.impl.schema.generic;
import java.util.List;
import java.util.stream.Collectors;
@@ -49,11 +49,6 @@ class GenericAvroRecord implements GenericRecord {
}
@Override
-public Object getField(Field field) {
-return getField(field.getName());
-}
-
-@Override
public Object getField(String fieldName) {
Object value = record.get(fieldName);
if (value instanceof Utf8) {
diff --git
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroSchema.java
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroSchema.java
similarity index 74%
rename from
pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroSchema.java
rename to
pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroSchema.java
index 4ccfe55..5fe4459 100644
---
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroSchema.java
+++
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroSchema.java
@@ -16,58 +16,40 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pulsar.client.impl.schema;
+package org.apache.pulsar.client.impl.schema.generic;
import static com.google.common.base.Preconditions.checkArgument;
-import static java.nio.charset.StandardCharsets.UTF_8;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
-import java.util.List;
-import java.util.stream.Collectors;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.
[GitHub] sijie closed pull request #2497: [schema] implement generic schema/record for Schema.JSON
sijie closed pull request #2497: [schema] implement generic schema/record for
Schema.JSON
URL: https://github.com/apache/incubator-pulsar/pull/2497
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
index 0a4fce43cb..46a49a1c39 100644
---
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
+++
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
@@ -38,7 +38,9 @@
* @param field the field to retrieve the value
* @return the value object
*/
-Object getField(Field field);
+default Object getField(Field field) {
+return getField(field.getName());
+}
/**
* Retrieve the value of the provided fieldName.
diff --git
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroRecord.java
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
similarity index 94%
rename from
pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroRecord.java
rename to
pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
index fb65c7aced..c9dbeb747b 100644
---
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroRecord.java
+++
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pulsar.client.impl.schema;
+package org.apache.pulsar.client.impl.schema.generic;
import java.util.List;
import java.util.stream.Collectors;
@@ -48,11 +48,6 @@
return fields;
}
-@Override
-public Object getField(Field field) {
-return getField(field.getName());
-}
-
@Override
public Object getField(String fieldName) {
Object value = record.get(fieldName);
diff --git
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroSchema.java
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroSchema.java
similarity index 74%
rename from
pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroSchema.java
rename to
pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroSchema.java
index 4ccfe55df0..5fe44596ba 100644
---
a/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/GenericAvroSchema.java
+++
b/pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroSchema.java
@@ -16,58 +16,40 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pulsar.client.impl.schema;
+package org.apache.pulsar.client.impl.schema.generic;
import static com.google.common.base.Preconditions.checkArgument;
-import static java.nio.charset.StandardCharsets.UTF_8;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
-import java.util.List;
-import java.util.stream.Collectors;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.Decoder;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.io.EncoderFactory;
-import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SchemaSerializationException;
-import org.apache.pulsar.client.api.schema.Field;
import org.apache.pulsar.client.api.schema.GenericRecord;
import org.apache.pulsar.common.schema.SchemaInfo;
/**
* A generic avro schema.
*/
-public class GenericAvroSchema implements Schema {
+class GenericAvroSchema extends GenericSchema {
-private final org.apache.avro.Schema schema;
-private final List fields;
-private final SchemaInfo schemaInfo;
private final GenericDatumWriter
datumWriter;
private BinaryEncoder encoder;
private final ByteArrayOutputStream byteArrayOutputStream;
private final GenericDatumReader
datumReader;
public GenericAvroSchema(SchemaInfo schemaInfo) {
-this.schemaInfo = schemaInfo;
-this.schema = new org.apache.avro.Schema.Parser().parse(
-new String(schemaInfo.getSchema(), UTF_8)
-);
-this.fields = schema.getFields()
-.stream()
-.map(f -> new Field(f.name(), f.pos()))
-.collect(Collectors.toList());
+sup
[GitHub] srkukarni opened a new pull request #2512: Misc Cleanups
srkukarni opened a new pull request #2512: Misc Cleanups URL: https://github.com/apache/incubator-pulsar/pull/2512 ### Motivation Explain here the context, and why you're making that change. What is the problem you're trying to solve. ### Modifications Describe the modifications you've done. ### Result After your change, what will change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new pull request #2513: [documentation] Add instructions for contributing c++/python code changes
sijie opened a new pull request #2513: [documentation] Add instructions for contributing c++/python code changes URL: https://github.com/apache/incubator-pulsar/pull/2513 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new pull request #2511: [documentation] Add section about licensing at contribution guide
sijie opened a new pull request #2511: [documentation] Add section about licensing at contribution guide URL: https://github.com/apache/incubator-pulsar/pull/2511 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2510: Handle /managed-ledgers znode existence on cluster init (#2379)
sijie commented on issue #2510: Handle /managed-ledgers znode existence on cluster init (#2379) URL: https://github.com/apache/incubator-pulsar/pull/2510#issuecomment-418531191 cherry-pick as bf4a50b in branch-2.1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch branch-2.1 updated: Handle /managed-ledgers znode existance on cluster init (#2379) (#2510)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new bf4a50b Handle /managed-ledgers znode existance on cluster init
(#2379) (#2510)
bf4a50b is described below
commit bf4a50b45dcaec7bd48b38a739bc4e7d43d36af1
Author: Ivan Kelly
AuthorDate: Tue Sep 4 23:41:47 2018 +0200
Handle /managed-ledgers znode existance on cluster init (#2379) (#2510)
ManagedLedger clients may create the /managed-ledger znode on boot, so
if a broker starts before the metadata is initialized, it could
potentially block initialization.
This patch changes this by making the existance of /managed-ledger a
non-error condition.
---
.../main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java| 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
index e35daef..9dd065a 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
@@ -147,7 +147,12 @@ public class PulsarClusterMetadataSetup {
}
}
-localZk.create("/managed-ledgers", new byte[0],
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+try {
+localZk.create("/managed-ledgers", new byte[0],
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+} catch (NodeExistsException e) {
+// Ignore
+}
+
localZk.create("/namespace", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
try {
[GitHub] sijie commented on issue #2025: Allow non-pesistent topics to be retrieved along with persistent ones from the "GetTopicsOfNamespace" method
sijie commented on issue #2025: Allow non-pesistent topics to be retrieved along with persistent ones from the "GetTopicsOfNamespace" method URL: https://github.com/apache/incubator-pulsar/pull/2025#issuecomment-418528212 @gordeevbr cool, thanks. I removed "[wip]". so other people know this PR is ready to review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] gordeevbr commented on issue #2025: [WiP] Allow non-pesistent topics to be retrieved along with persistent ones from the "GetTopicsOfNamespace" method
gordeevbr commented on issue #2025: [WiP] Allow non-pesistent topics to be retrieved along with persistent ones from the "GetTopicsOfNamespace" method URL: https://github.com/apache/incubator-pulsar/pull/2025#issuecomment-418527695 @jiazhai @merlimat @sijie I have fixed old tests that were failing, added some new tests to test suites where it was possible, updated features to be consistent with HTTP Lookup, fixed all found issues, and updated with master branch. There's not much new in these commits, mostly fixes and tests. I think it should be ready now. I am, of course, ready to update this PR if deemed necessary. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Handle /managed-ledgers znode existance on cluster init (#2379) (#2510)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 889b1bf Handle /managed-ledgers znode existance on cluster init
(#2379) (#2510)
889b1bf is described below
commit 889b1bfe8c28075d64ee02bd63510de0d01772b9
Author: Ivan Kelly
AuthorDate: Tue Sep 4 23:41:47 2018 +0200
Handle /managed-ledgers znode existance on cluster init (#2379) (#2510)
ManagedLedger clients may create the /managed-ledger znode on boot, so
if a broker starts before the metadata is initialized, it could
potentially block initialization.
This patch changes this by making the existance of /managed-ledger a
non-error condition.
---
.../main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java| 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
index e35daef..9dd065a 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
@@ -147,7 +147,12 @@ public class PulsarClusterMetadataSetup {
}
}
-localZk.create("/managed-ledgers", new byte[0],
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+try {
+localZk.create("/managed-ledgers", new byte[0],
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+} catch (NodeExistsException e) {
+// Ignore
+}
+
localZk.create("/namespace", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
try {
[GitHub] merlimat closed pull request #2510: Handle /managed-ledgers znode existence on cluster init (#2379)
merlimat closed pull request #2510: Handle /managed-ledgers znode existence on
cluster init (#2379)
URL: https://github.com/apache/incubator-pulsar/pull/2510
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
index e35daef3e2..9dd065a42a 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
@@ -147,7 +147,12 @@ public static void main(String[] args) throws Exception {
}
}
-localZk.create("/managed-ledgers", new byte[0],
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+try {
+localZk.create("/managed-ledgers", new byte[0],
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+} catch (NodeExistsException e) {
+// Ignore
+}
+
localZk.create("/namespace", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
try {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sijie commented on issue #2471: PIP-23: Pulsar Java Client Interceptors.
sijie commented on issue #2471: PIP-23: Pulsar Java Client Interceptors. URL: https://github.com/apache/incubator-pulsar/pull/2471#issuecomment-418527004 @codelipenghui I fixed ReplicatorTest and V1_ReplicatorTest. so the CI should be passing now. once the CI is passed, will merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2340: Cherry-pick GCS offloader documentation to new website
sijie commented on issue #2340: Cherry-pick GCS offloader documentation to new website URL: https://github.com/apache/incubator-pulsar/issues/2340#issuecomment-418522474 This is fixed by #2364 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed issue #2340: Cherry-pick GCS offloader documentation to new website
sijie closed issue #2340: Cherry-pick GCS offloader documentation to new website URL: https://github.com/apache/incubator-pulsar/issues/2340 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2508: PIP-22: Dead Letter Topic
sijie commented on issue #2508: PIP-22: Dead Letter Topic URL: https://github.com/apache/incubator-pulsar/pull/2508#issuecomment-418506210 @rdhabalia @merlimat can you help reviewing @codelipenghui 's pull request? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2508: PIP-22: Dead Letter Topic
sijie commented on issue #2508: PIP-22: Dead Letter Topic URL: https://github.com/apache/incubator-pulsar/pull/2508#issuecomment-418505462 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2500: [schema] add schemas for primtive types
sijie commented on issue #2500: [schema] add schemas for primtive types URL: https://github.com/apache/incubator-pulsar/pull/2500#issuecomment-418505286 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2503: add auto ack and timeout configurable
sijie commented on issue #2503: add auto ack and timeout configurable URL: https://github.com/apache/incubator-pulsar/pull/2503#issuecomment-418505083 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2409: Added HDFS Sink
sijie commented on issue #2409: Added HDFS Sink URL: https://github.com/apache/incubator-pulsar/pull/2409#issuecomment-418504900 @srkukarni @merlimat can you review this PR and make sure we can land this in 2.2 release? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #2440: Issue 2313: create a JDBC sink connector
sijie closed pull request #2440: Issue 2313: create a JDBC sink connector
URL: https://github.com/apache/incubator-pulsar/pull/2440
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/distribution/io/src/assemble/io.xml
b/distribution/io/src/assemble/io.xml
index 8cf7fce208..bb75e84e34 100644
--- a/distribution/io/src/assemble/io.xml
+++ b/distribution/io/src/assemble/io.xml
@@ -74,6 +74,11 @@
connectors
644
+
+
${basedir}/../../pulsar-io/jdbc/target/pulsar-io-jdbc-${project.version}.nar
+ connectors
+ 644
+
${basedir}/../../pulsar-io/data-genenator/target/pulsar-io-data-generator-${project.version}.nar
connectors
diff --git a/pom.xml b/pom.xml
index 4bb3cf63bb..c0dc20d492 100644
--- a/pom.xml
+++ b/pom.xml
@@ -166,6 +166,8 @@ flexible messaging model and an intuitive client
API.
1.11.297
1.8.2
2.1.1
+3.8.11.2
+8.0.11
0.206
@@ -818,6 +820,11 @@ flexible messaging model and an intuitive client
API.
kafka
${testcontainers.version}
+
+org.testcontainers
+mysql
+${testcontainers.version}
+
org.arquillian.cube
arquillian-cube-docker
@@ -1083,7 +1090,7 @@ flexible messaging model and an intuitive client
API.
src/main/java/org/apache/pulsar/common/api/proto/PulsarApi.java
src/main/java/org/apache/pulsar/broker/service/schema/proto/SchemaRegistryFormat.java
bin/proto/MLDataFormats_pb2.py
-
+
src/main/java/org/apache/pulsar/io/kinesis/fbs/CompressionType.java
src/main/java/org/apache/pulsar/io/kinesis/fbs/EncryptionCtx.java
diff --git a/pulsar-io/jdbc/lombok.config b/pulsar-io/jdbc/lombok.config
new file mode 100644
index 00..9a9adee272
--- /dev/null
+++ b/pulsar-io/jdbc/lombok.config
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+## This file is to fix the conflict with jackson error like this:
+##com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
construct instance of ...
+lombok.anyConstructor.addConstructorProperties=true
+config.stopBubbling = true
diff --git a/pulsar-io/jdbc/pom.xml b/pulsar-io/jdbc/pom.xml
new file mode 100644
index 00..eed85883ec
--- /dev/null
+++ b/pulsar-io/jdbc/pom.xml
@@ -0,0 +1,96 @@
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+org.apache.pulsar
+pulsar-io
+2.2.0-incubating-SNAPSHOT
+
+
+ pulsar-io-jdbc
+ Pulsar IO :: Jdbc
+
+
+
+
+ ${project.groupId}
+ pulsar-io-core
+ ${project.version}
+
+
+
+
+ org.apache.avro
+ avro
+ ${avro.version}
+
+
+
+ ${project.groupId}
+ pulsar-functions-instance
+ ${project.version}
+
+
+
+ org.xerial
+ sqlite-jdbc
+ ${sqlite-jdbc.version}
+ runtime
+
+
+
+ mysql
+ mysql-connector-java
+ ${mysql-jdbc.version}
+ runtime
+
+
+
+ com.fasterxml.jackson.core
+ jackson-databind
+
+
+
+ com.fasterxml.jackson.dataformat
+ jackson-dataformat-yaml
+
+
+
+ ${project.groupId}
+ pulsar-client-original
+ ${project.version}
+
+
+
+
+
+
+
+org.apache.nifi
+nifi-nar-maven-plugin
+
+
+
+
+
+
diff --git
a/pulsar-io/jdbc/src/main/java/org/apache/pulsar/io/jdbc/JdbcAbstractSink.java
b/pulsar-io/jdbc/src/main/java/org/apache/pulsar/io/jdbc/JdbcAbstractSink.java
new file mode 100644
index 00..425fb57ac1
--- /dev/null
+++
b/pulsar-io/jdbc/src/main/java/org/apache/pulsar/io/jdbc/JdbcAbstractSink.java
@@ -0,0 +1,197 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE fi
[incubator-pulsar] branch master updated: Issue 2313: create a JDBC sink connector (#2440)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 79ac46a Issue 2313: create a JDBC sink connector (#2440)
79ac46a is described below
commit 79ac46a6ae50e48bed4ccb680d1f7945611f0565
Author: Jia Zhai
AuthorDate: Wed Sep 5 04:20:42 2018 +0800
Issue 2313: create a JDBC sink connector (#2440)
### Motivation
This change is trying to add a basic JDBC sink connector.
### Modifications
Add the jdbc module to the pulsar-io sub-module.
Add unit test and integration test for it.
### Result
ut and integration test pass.
Master Issue: #2442
---
distribution/io/src/assemble/io.xml| 5 +
pom.xml| 9 +-
pulsar-io/jdbc/lombok.config | 23 +++
pulsar-io/jdbc/pom.xml | 96 ++
.../apache/pulsar/io/jdbc/JdbcAbstractSink.java| 197 +
.../apache/pulsar/io/jdbc/JdbcAvroSchemaSink.java | 92 ++
.../org/apache/pulsar/io/jdbc/JdbcSinkConfig.java | 63 +++
.../java/org/apache/pulsar/io/jdbc/JdbcUtils.java | 178 +++
.../resources/META-INF/services/pulsar-io.yaml | 22 +++
.../org/apache/pulsar/io/jdbc/JdbcSinkTest.java| 135 ++
.../org/apache/pulsar/io/jdbc/JdbcUtilsTest.java | 95 ++
.../org/apache/pulsar/io/jdbc/SqliteUtils.java | 111
pulsar-io/pom.xml | 1 +
tests/integration/pom.xml | 21 +++
.../integration/functions/PulsarFunctionsTest.java | 54 +-
.../functions/PulsarFunctionsTestBase.java | 2 +-
.../tests/integration/io/JdbcSinkTester.java | 137 ++
.../tests/integration/suites/PulsarTestSuite.java | 11 ++
18 files changed, 1245 insertions(+), 7 deletions(-)
diff --git a/distribution/io/src/assemble/io.xml
b/distribution/io/src/assemble/io.xml
index 8cf7fce..bb75e84 100644
--- a/distribution/io/src/assemble/io.xml
+++ b/distribution/io/src/assemble/io.xml
@@ -75,6 +75,11 @@
644
+
${basedir}/../../pulsar-io/jdbc/target/pulsar-io-jdbc-${project.version}.nar
+ connectors
+ 644
+
+
${basedir}/../../pulsar-io/data-genenator/target/pulsar-io-data-generator-${project.version}.nar
connectors
644
diff --git a/pom.xml b/pom.xml
index 1ed2802..681f18e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -167,6 +167,8 @@ flexible messaging model and an intuitive client
API.
1.11.297
1.8.2
2.1.1
+3.8.11.2
+8.0.11
0.206
1.6.0
2.11
@@ -822,6 +824,11 @@ flexible messaging model and an intuitive client
API.
${testcontainers.version}
+org.testcontainers
+mysql
+${testcontainers.version}
+
+
org.arquillian.cube
arquillian-cube-docker
${arquillian-cube.version}
@@ -1086,7 +1093,7 @@ flexible messaging model and an intuitive client
API.
src/main/java/org/apache/pulsar/common/api/proto/PulsarApi.java
src/main/java/org/apache/pulsar/broker/service/schema/proto/SchemaRegistryFormat.java
bin/proto/MLDataFormats_pb2.py
-
+
src/main/java/org/apache/pulsar/io/kinesis/fbs/CompressionType.java
src/main/java/org/apache/pulsar/io/kinesis/fbs/EncryptionCtx.java
diff --git a/pulsar-io/jdbc/lombok.config b/pulsar-io/jdbc/lombok.config
new file mode 100644
index 000..9a9adee
--- /dev/null
+++ b/pulsar-io/jdbc/lombok.config
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+## This file is to fix the conflict with jackson error like this:
+##com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
construct instance of ...
+lombok.anyConstructor.addConstructorProperties=true
+config.stopBubbling = true
diff --git a/pulsar-io/jdbc/pom.xml b/pulsar-io/jdbc/pom.xml
new file mode 100644
index 000..eed8588
-
[GitHub] sijie commented on issue #2509: Don't initialize /managed-ledgers on client creation (#2379)
sijie commented on issue #2509: Don't initialize /managed-ledgers on client creation (#2379) URL: https://github.com/apache/incubator-pulsar/pull/2509#issuecomment-418504238 cherry-pick as 8a4f5cc in branch-2.1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2510: Handle /managed-ledgers znode existence on cluster init (#2379)
sijie commented on issue #2510: Handle /managed-ledgers znode existence on cluster init (#2379) URL: https://github.com/apache/incubator-pulsar/pull/2510#issuecomment-418504311 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch branch-2.1 updated: Don't initialize /managed-ledgers on client creation (#2379) (#2509)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 8a4f5cc Don't initialize /managed-ledgers on client creation (#2379)
(#2509)
8a4f5cc is described below
commit 8a4f5ccad08522a5774dbc0500b3f48ea73a6fb6
Author: Ivan Kelly
AuthorDate: Tue Sep 4 22:18:46 2018 +0200
Don't initialize /managed-ledgers on client creation (#2379) (#2509)
Normally the /managed-ledgers znode is created by the
initialize-cluster-metadata command when a cluster is being turned
up.
However, the ManagedLedger client also creates it on boot. This has
caused issues in the past, where if a broker is started before
initialize-cluster-metadata is run, then initialize-cluster-metadata
fails because it sees the /managed-ledger znode.
This patch removes the automatic creation of this znode from the
client boot process.
---
.../mledger/impl/MetaStoreImplZookeeper.java | 64 +++---
.../mledger/impl/MetaStoreImplZookeeperTest.java | 52 ++
.../bookkeeper/test/BookKeeperClusterTestCase.java | 4 ++
.../bookkeeper/test/MockedBookKeeperTestCase.java | 4 ++
4 files changed, 117 insertions(+), 7 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
index 354c04f..4d90b91 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
@@ -25,9 +25,11 @@ import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.TextFormat;
import com.google.protobuf.TextFormat.ParseException;
+import java.io.File;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
+import java.util.function.Consumer;
import org.apache.bookkeeper.common.util.OrderedExecutor;
import org.apache.bookkeeper.mledger.ManagedLedgerException;
@@ -35,7 +37,6 @@ import
org.apache.bookkeeper.mledger.ManagedLedgerException.BadVersionException;
import org.apache.bookkeeper.mledger.ManagedLedgerException.MetaStoreException;
import org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedCursorInfo;
import org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo;
-import org.apache.bookkeeper.util.ZkUtils;
import org.apache.zookeeper.AsyncCallback.StringCallback;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
@@ -95,10 +96,6 @@ public class MetaStoreImplZookeeper implements MetaStore {
throws Exception {
this.zk = zk;
this.executor = executor;
-
-if (zk.exists(prefixName, false) == null) {
-zk.create(prefixName, new byte[0], Acl, CreateMode.PERSISTENT);
-}
}
//
@@ -157,8 +154,8 @@ public class MetaStoreImplZookeeper implements MetaStore {
}
};
-ZkUtils.asyncCreateFullPathOptimistic(zk, prefix +
ledgerName, new byte[0], Acl,
-CreateMode.PERSISTENT, createcb, null);
+asyncCreateFullPathOptimistic(zk, prefixName,
ledgerName, new byte[0], Acl,
+
CreateMode.PERSISTENT, createcb);
} else {
// Tried to open a managed ledger but it doesn't
exist and we shouldn't creating it at this
// point
@@ -365,5 +362,58 @@ public class MetaStoreImplZookeeper implements MetaStore {
}
+public static void asyncCreateFullPathOptimistic(
+final ZooKeeper zk, final String basePath, final String nodePath,
final byte[] data,
+final List acl, final CreateMode createMode, final
StringCallback callback) {
+String fullPath = basePath + "/" + nodePath;
+
+zk.create(fullPath, data, acl, createMode,
+ (rc, path, ignoreCtx1, name) -> {
+ Runnable retry = () -> {
+ asyncCreateFullPathOptimistic(zk, basePath,
nodePath, data,
+acl, createMode,
callback);
+ };
+
+ Consumer complete = (finalrc) -> {
+ callback.processResult(finalrc, path, null, name);
+ };
+
+ if (rc != Code.NONODE.intValue()) {
+ complete.accept(rc);
+ return;
+ }
+
+ // Since I got a nonode, it means t
[incubator-pulsar] branch master updated: Don't initialize /managed-ledgers on client creation (#2379) (#2509)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 8763c74 Don't initialize /managed-ledgers on client creation (#2379)
(#2509)
8763c74 is described below
commit 8763c746b5b38901d7fbb77e5766f55c563ba25e
Author: Ivan Kelly
AuthorDate: Tue Sep 4 22:18:46 2018 +0200
Don't initialize /managed-ledgers on client creation (#2379) (#2509)
Normally the /managed-ledgers znode is created by the
initialize-cluster-metadata command when a cluster is being turned
up.
However, the ManagedLedger client also creates it on boot. This has
caused issues in the past, where if a broker is started before
initialize-cluster-metadata is run, then initialize-cluster-metadata
fails because it sees the /managed-ledger znode.
This patch removes the automatic creation of this znode from the
client boot process.
---
.../mledger/impl/MetaStoreImplZookeeper.java | 64 +++---
.../mledger/impl/MetaStoreImplZookeeperTest.java | 52 ++
.../bookkeeper/test/BookKeeperClusterTestCase.java | 4 ++
.../bookkeeper/test/MockedBookKeeperTestCase.java | 4 ++
4 files changed, 117 insertions(+), 7 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
index 354c04f..4d90b91 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
@@ -25,9 +25,11 @@ import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.TextFormat;
import com.google.protobuf.TextFormat.ParseException;
+import java.io.File;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
+import java.util.function.Consumer;
import org.apache.bookkeeper.common.util.OrderedExecutor;
import org.apache.bookkeeper.mledger.ManagedLedgerException;
@@ -35,7 +37,6 @@ import
org.apache.bookkeeper.mledger.ManagedLedgerException.BadVersionException;
import org.apache.bookkeeper.mledger.ManagedLedgerException.MetaStoreException;
import org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedCursorInfo;
import org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo;
-import org.apache.bookkeeper.util.ZkUtils;
import org.apache.zookeeper.AsyncCallback.StringCallback;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
@@ -95,10 +96,6 @@ public class MetaStoreImplZookeeper implements MetaStore {
throws Exception {
this.zk = zk;
this.executor = executor;
-
-if (zk.exists(prefixName, false) == null) {
-zk.create(prefixName, new byte[0], Acl, CreateMode.PERSISTENT);
-}
}
//
@@ -157,8 +154,8 @@ public class MetaStoreImplZookeeper implements MetaStore {
}
};
-ZkUtils.asyncCreateFullPathOptimistic(zk, prefix +
ledgerName, new byte[0], Acl,
-CreateMode.PERSISTENT, createcb, null);
+asyncCreateFullPathOptimistic(zk, prefixName,
ledgerName, new byte[0], Acl,
+
CreateMode.PERSISTENT, createcb);
} else {
// Tried to open a managed ledger but it doesn't
exist and we shouldn't creating it at this
// point
@@ -365,5 +362,58 @@ public class MetaStoreImplZookeeper implements MetaStore {
}
+public static void asyncCreateFullPathOptimistic(
+final ZooKeeper zk, final String basePath, final String nodePath,
final byte[] data,
+final List acl, final CreateMode createMode, final
StringCallback callback) {
+String fullPath = basePath + "/" + nodePath;
+
+zk.create(fullPath, data, acl, createMode,
+ (rc, path, ignoreCtx1, name) -> {
+ Runnable retry = () -> {
+ asyncCreateFullPathOptimistic(zk, basePath,
nodePath, data,
+acl, createMode,
callback);
+ };
+
+ Consumer complete = (finalrc) -> {
+ callback.processResult(finalrc, path, null, name);
+ };
+
+ if (rc != Code.NONODE.intValue()) {
+ complete.accept(rc);
+ return;
+ }
+
+ // Since I got a nonode, it means that my p
[GitHub] sijie closed pull request #2509: Don't initialize /managed-ledgers on client creation (#2379)
sijie closed pull request #2509: Don't initialize /managed-ledgers on client
creation (#2379)
URL: https://github.com/apache/incubator-pulsar/pull/2509
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
index 354c04f4fc..4d90b91371 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImplZookeeper.java
@@ -25,9 +25,11 @@
import com.google.protobuf.TextFormat;
import com.google.protobuf.TextFormat.ParseException;
+import java.io.File;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
+import java.util.function.Consumer;
import org.apache.bookkeeper.common.util.OrderedExecutor;
import org.apache.bookkeeper.mledger.ManagedLedgerException;
@@ -35,7 +37,6 @@
import org.apache.bookkeeper.mledger.ManagedLedgerException.MetaStoreException;
import org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedCursorInfo;
import org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo;
-import org.apache.bookkeeper.util.ZkUtils;
import org.apache.zookeeper.AsyncCallback.StringCallback;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
@@ -95,10 +96,6 @@ public MetaStoreImplZookeeper(ZooKeeper zk, OrderedExecutor
executor)
throws Exception {
this.zk = zk;
this.executor = executor;
-
-if (zk.exists(prefixName, false) == null) {
-zk.create(prefixName, new byte[0], Acl, CreateMode.PERSISTENT);
-}
}
//
@@ -157,8 +154,8 @@ public void getManagedLedgerInfo(final String ledgerName,
boolean createIfMissin
}
};
-ZkUtils.asyncCreateFullPathOptimistic(zk, prefix +
ledgerName, new byte[0], Acl,
-CreateMode.PERSISTENT, createcb, null);
+asyncCreateFullPathOptimistic(zk, prefixName,
ledgerName, new byte[0], Acl,
+
CreateMode.PERSISTENT, createcb);
} else {
// Tried to open a managed ledger but it doesn't
exist and we shouldn't creating it at this
// point
@@ -365,5 +362,58 @@ private ManagedCursorInfo parseManagedCursorInfo(byte[]
data)
}
+public static void asyncCreateFullPathOptimistic(
+final ZooKeeper zk, final String basePath, final String nodePath,
final byte[] data,
+final List acl, final CreateMode createMode, final
StringCallback callback) {
+String fullPath = basePath + "/" + nodePath;
+
+zk.create(fullPath, data, acl, createMode,
+ (rc, path, ignoreCtx1, name) -> {
+ Runnable retry = () -> {
+ asyncCreateFullPathOptimistic(zk, basePath,
nodePath, data,
+acl, createMode,
callback);
+ };
+
+ Consumer complete = (finalrc) -> {
+ callback.processResult(finalrc, path, null, name);
+ };
+
+ if (rc != Code.NONODE.intValue()) {
+ complete.accept(rc);
+ return;
+ }
+
+ // Since I got a nonode, it means that my parents don't
exist
+ // create mode is persistent since ephemeral nodes can't
be
+ // parents
+ String nodeParent = new File(nodePath).getParent();
+ if (nodeParent == null) {
+ zk.exists(basePath, false,
+(existsRc, existsPath, ignoreCtx2, stat)
-> {
+if (existsRc == Code.OK.intValue()) {
+if (stat != null) {
+retry.run();
+} else {
+
complete.accept(Code.NONODE.intValue());
+}
+} else {
+complete.accept(existsRc);
+}
+}, null);
+ } else {
+
[GitHub] sijie commented on issue #2487: Ensure standalone service comes back quickly after ungraceful restarts
sijie commented on issue #2487: Ensure standalone service comes back quickly after ungraceful restarts URL: https://github.com/apache/incubator-pulsar/pull/2487#issuecomment-418494023 cherry-picked as 113418d in branch-2.1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch branch-2.1 updated: Ensure standalone service comes back quickly after ungraceful restarts (#2487)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 113418d Ensure standalone service comes back quickly after ungraceful
restarts (#2487)
113418d is described below
commit 113418d3e7b4d02597bed7a939a46e5f66829621
Author: Matteo Merli
AuthorDate: Tue Sep 4 12:34:48 2018 -0700
Ensure standalone service comes back quickly after ungraceful restarts
(#2487)
* Ensure standalone service comes back quickly after ungraceful restarts
* Handle NoNode errors when deleting
* Added NoopLoadManager for standalone mode
---
conf/standalone.conf | 2 +
.../pulsar/broker/loadbalance/NoopLoadManager.java | 158 +
.../loadbalance/impl/ModularLoadManagerImpl.java | 3 +-
.../pulsar/zookeeper/LocalBookkeeperEnsemble.java | 15 +-
4 files changed, 176 insertions(+), 2 deletions(-)
diff --git a/conf/standalone.conf b/conf/standalone.conf
index 74a5702..755b76e 100644
--- a/conf/standalone.conf
+++ b/conf/standalone.conf
@@ -311,6 +311,8 @@ autoSkipNonRecoverableData=false
### --- Load balancer --- ###
+loadManagerClassName=org.apache.pulsar.broker.loadbalance.NoopLoadManager
+
# Enable load balancer
loadBalancerEnabled=false
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java
new file mode 100644
index 000..5773c61
--- /dev/null
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java
@@ -0,0 +1,158 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.bookkeeper.util.ZkUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.impl.PulsarResourceDescription;
+import org.apache.pulsar.broker.loadbalance.impl.SimpleResourceUnit;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.stats.Metrics;
+import org.apache.pulsar.common.util.ObjectMapperFactory;
+import org.apache.pulsar.policies.data.loadbalancer.LoadManagerReport;
+import org.apache.pulsar.policies.data.loadbalancer.LocalBrokerData;
+import org.apache.pulsar.policies.data.loadbalancer.ServiceLookupData;
+import org.apache.pulsar.zookeeper.ZooKeeperCache.Deserializer;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooKeeper;
+
+public class NoopLoadManager implements LoadManager {
+
+private String lookupServiceAddress;
+private ResourceUnit localResourceUnit;
+private ZooKeeper zkClient;
+
+LocalBrokerData localData;
+
+private static final Deserializer loadReportDeserializer
= (key, content) -> ObjectMapperFactory
+.getThreadLocal()
+.readValue(content, LocalBrokerData.class);
+
+@Override
+public void initialize(PulsarService pulsar) {
+lookupServiceAddress = pulsar.getAdvertisedAddress() + ":" +
pulsar.getConfiguration().getWebServicePort();
+localResourceUnit = new SimpleResourceUnit(String.format("http://%s";,
lookupServiceAddress),
+new PulsarResourceDescription());
+zkClient = pulsar.getZkClient();
+
+localData = new LocalBrokerData(pulsar.getWebServiceAddress(),
pulsar.getWebServiceAddressTls(),
+pulsar.getBrokerServiceUrl(), pulsar.getBrokerServiceUrlTls());
+}
+
+@Override
+public void start() throws PulsarServerException {
+String brokerZnodePath = LoadManager.LOADBALANCE_BROKERS_ROOT + "/" +
lookupServiceAddress;
+
+try {
+// When running in standalone, this error can happen when killing
the "standalone" process
+/
[incubator-pulsar] branch master updated: Ensure standalone service comes back quickly after ungraceful restarts (#2487)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 03d1302 Ensure standalone service comes back quickly after ungraceful
restarts (#2487)
03d1302 is described below
commit 03d1302c91506df1dace73a60689e78ea589505f
Author: Matteo Merli
AuthorDate: Tue Sep 4 12:34:48 2018 -0700
Ensure standalone service comes back quickly after ungraceful restarts
(#2487)
* Ensure standalone service comes back quickly after ungraceful restarts
* Handle NoNode errors when deleting
* Added NoopLoadManager for standalone mode
---
conf/standalone.conf | 2 +
.../pulsar/broker/loadbalance/NoopLoadManager.java | 158 +
.../loadbalance/impl/ModularLoadManagerImpl.java | 3 +-
.../pulsar/zookeeper/LocalBookkeeperEnsemble.java | 15 +-
4 files changed, 176 insertions(+), 2 deletions(-)
diff --git a/conf/standalone.conf b/conf/standalone.conf
index b09fcd6..a68664c 100644
--- a/conf/standalone.conf
+++ b/conf/standalone.conf
@@ -320,6 +320,8 @@ autoSkipNonRecoverableData=false
### --- Load balancer --- ###
+loadManagerClassName=org.apache.pulsar.broker.loadbalance.NoopLoadManager
+
# Enable load balancer
loadBalancerEnabled=false
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java
new file mode 100644
index 000..5773c61
--- /dev/null
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java
@@ -0,0 +1,158 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.bookkeeper.util.ZkUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.impl.PulsarResourceDescription;
+import org.apache.pulsar.broker.loadbalance.impl.SimpleResourceUnit;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.stats.Metrics;
+import org.apache.pulsar.common.util.ObjectMapperFactory;
+import org.apache.pulsar.policies.data.loadbalancer.LoadManagerReport;
+import org.apache.pulsar.policies.data.loadbalancer.LocalBrokerData;
+import org.apache.pulsar.policies.data.loadbalancer.ServiceLookupData;
+import org.apache.pulsar.zookeeper.ZooKeeperCache.Deserializer;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooKeeper;
+
+public class NoopLoadManager implements LoadManager {
+
+private String lookupServiceAddress;
+private ResourceUnit localResourceUnit;
+private ZooKeeper zkClient;
+
+LocalBrokerData localData;
+
+private static final Deserializer loadReportDeserializer
= (key, content) -> ObjectMapperFactory
+.getThreadLocal()
+.readValue(content, LocalBrokerData.class);
+
+@Override
+public void initialize(PulsarService pulsar) {
+lookupServiceAddress = pulsar.getAdvertisedAddress() + ":" +
pulsar.getConfiguration().getWebServicePort();
+localResourceUnit = new SimpleResourceUnit(String.format("http://%s";,
lookupServiceAddress),
+new PulsarResourceDescription());
+zkClient = pulsar.getZkClient();
+
+localData = new LocalBrokerData(pulsar.getWebServiceAddress(),
pulsar.getWebServiceAddressTls(),
+pulsar.getBrokerServiceUrl(), pulsar.getBrokerServiceUrlTls());
+}
+
+@Override
+public void start() throws PulsarServerException {
+String brokerZnodePath = LoadManager.LOADBALANCE_BROKERS_ROOT + "/" +
lookupServiceAddress;
+
+try {
+// When running in standalone, this error can happen when killing
the "standalone" process
+// ungra
[GitHub] merlimat closed pull request #2487: Ensure standalone service comes back quickly after ungraceful restarts
merlimat closed pull request #2487: Ensure standalone service comes back
quickly after ungraceful restarts
URL: https://github.com/apache/incubator-pulsar/pull/2487
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/conf/standalone.conf b/conf/standalone.conf
index 09d369c6ff..f9aad4c76f 100644
--- a/conf/standalone.conf
+++ b/conf/standalone.conf
@@ -320,6 +320,8 @@ autoSkipNonRecoverableData=false
### --- Load balancer --- ###
+loadManagerClassName=org.apache.pulsar.broker.loadbalance.NoopLoadManager
+
# Enable load balancer
loadBalancerEnabled=false
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java
new file mode 100644
index 00..5773c61ee1
--- /dev/null
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java
@@ -0,0 +1,158 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.bookkeeper.util.ZkUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.impl.PulsarResourceDescription;
+import org.apache.pulsar.broker.loadbalance.impl.SimpleResourceUnit;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.stats.Metrics;
+import org.apache.pulsar.common.util.ObjectMapperFactory;
+import org.apache.pulsar.policies.data.loadbalancer.LoadManagerReport;
+import org.apache.pulsar.policies.data.loadbalancer.LocalBrokerData;
+import org.apache.pulsar.policies.data.loadbalancer.ServiceLookupData;
+import org.apache.pulsar.zookeeper.ZooKeeperCache.Deserializer;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooKeeper;
+
+public class NoopLoadManager implements LoadManager {
+
+private String lookupServiceAddress;
+private ResourceUnit localResourceUnit;
+private ZooKeeper zkClient;
+
+LocalBrokerData localData;
+
+private static final Deserializer loadReportDeserializer
= (key, content) -> ObjectMapperFactory
+.getThreadLocal()
+.readValue(content, LocalBrokerData.class);
+
+@Override
+public void initialize(PulsarService pulsar) {
+lookupServiceAddress = pulsar.getAdvertisedAddress() + ":" +
pulsar.getConfiguration().getWebServicePort();
+localResourceUnit = new SimpleResourceUnit(String.format("http://%s";,
lookupServiceAddress),
+new PulsarResourceDescription());
+zkClient = pulsar.getZkClient();
+
+localData = new LocalBrokerData(pulsar.getWebServiceAddress(),
pulsar.getWebServiceAddressTls(),
+pulsar.getBrokerServiceUrl(), pulsar.getBrokerServiceUrlTls());
+}
+
+@Override
+public void start() throws PulsarServerException {
+String brokerZnodePath = LoadManager.LOADBALANCE_BROKERS_ROOT + "/" +
lookupServiceAddress;
+
+try {
+// When running in standalone, this error can happen when killing
the "standalone" process
+// ungracefully since the ZK session will not be closed and it
will take some time for ZK server
+// to prune the expired sessions after startup.
+// Since there's a single broker instance running, it's safe, in
this mode, to remove the old lock
+
+// Delete and recreate z-node
+try {
+if (zkClient.exists(brokerZnodePath, null) != null) {
+zkClient.delete(brokerZnodePath, -1);
+}
+} catch (NoNodeException nne) {
+// Ignore if z-node was just expired
+}
+
+ZkUtils.create
[GitHub] sijie commented on issue #2440: Issue 2313: create a JDBC sink connector
sijie commented on issue #2440: Issue 2313: create a JDBC sink connector URL: https://github.com/apache/incubator-pulsar/pull/2440#issuecomment-418480681 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #2434: Add pulsar flink sink connector
sijie closed pull request #2434: Add pulsar flink sink connector
URL: https://github.com/apache/incubator-pulsar/pull/2434
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/pom.xml b/pom.xml
index 4bb3cf63bb..1ed2802fda 100644
--- a/pom.xml
+++ b/pom.xml
@@ -92,6 +92,7 @@ flexible messaging model and an intuitive client
API.
pulsar-proxy
pulsar-discovery-service
pulsar-storm
+pulsar-flink
pulsar-spark
pulsar-zookeeper-utils
pulsar-testclient
@@ -167,6 +168,8 @@ flexible messaging model and an intuitive client
API.
1.8.2
2.1.1
0.206
+1.6.0
+2.11
1.15.1
diff --git a/pulsar-flink/pom.xml b/pulsar-flink/pom.xml
new file mode 100644
index 00..97ab4fb66e
--- /dev/null
+++ b/pulsar-flink/pom.xml
@@ -0,0 +1,137 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";
+ xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>
+ 4.0.0
+
+
+org.apache.pulsar
+pulsar
+2.2.0-incubating-SNAPSHOT
+..
+
+
+ pulsar-flink
+ Pulsar Flink Connectors
+
+
+
+
+ org.apache.flink
+ flink-streaming-java_${scala.binary.version}
+ ${flink.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-table_${scala.binary.version}
+ ${flink.version}
+ provided
+
+ true
+
+
+
+ ${project.groupId}
+ pulsar-client
+ ${project.version}
+
+
+
+ org.apache.flink
+ flink-runtime_${scala.binary.version}
+ ${flink.version}
+ test-jar
+ test
+
+
+
+ org.apache.flink
+ flink-tests_${scala.binary.version}
+ ${flink.version}
+ test
+ test-jar
+
+
+
+ org.apache.flink
+ flink-streaming-java_${scala.binary.version}
+ ${flink.version}
+ test-jar
+ test
+
+
+
+ org.apache.flink
+ flink-test-utils_${scala.binary.version}
+ ${flink.version}
+ test
+
+
+
+ org.javassist
+ javassist
+ 3.20.0-GA
+ test
+
+
+
+
+
+
+
+src/main/resources
+true
+
+
+
+
+org.apache.maven.plugins
+maven-shade-plugin
+
+
+package
+
+ shade
+
+
+ true
+
true
+
+
+ com.google.guava:guava
+
+
+
+
+ com.google
+ pulsar-flink-shade.com.google
+
+
+
+
+
+
+
+
+
diff --git
a/pulsar-flink/src/main/java/org/apache/flink/streaming/connectors/pulsar/FlinkPulsarProducer.java
b/pulsar-flink/src/main/java/org/apache/flink/streaming/connectors/pulsar/FlinkPulsarProducer.java
new file mode 100644
index 00..bddfee43ef
--- /dev/null
+++
b/pulsar-flink/src/main/java/org/apache/flink/streaming/connectors/pulsar/FlinkPulsarProducer.java
@@ -0,0 +1,314 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.flink.streaming.connectors.pulsar;
+
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.sink.RichSinkFunction;
+import org.apache.flink.streaming.api.operators.StreamingRuntimeContext;
+import
org.apache.flink.streaming.connectors.pulsar.partitioner.PulsarKeyExtractor;
+import org.apache.flink.util.SerializableObject;
+import org.apache.pulsar.client.api
[incubator-pulsar] branch master updated: Add pulsar flink sink connector (#2434)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new ad03a07 Add pulsar flink sink connector (#2434)
ad03a07 is described below
commit ad03a07d5f5d93654f1d5afb8694989db5688095
Author: Ali Ahmed
AuthorDate: Tue Sep 4 11:59:03 2018 -0700
Add pulsar flink sink connector (#2434)
@XiaoZYang introduces a module for a pulsar sink connector for flink. This
PR is moving the work from flink repo to pulsar repo, so the flink connector
can be released faster along with Pulsar releases.
Original Flink Github Issue: apache/flink#5845
Jira Issue: https://issues.apache.org/jira/browse/FLINK-9168
Original Author: @XiaoZYang (Zong Yang Xiao)
---
pom.xml| 3 +
pulsar-flink/pom.xml | 137 +
.../connectors/pulsar/FlinkPulsarProducer.java | 314 +
.../connectors/pulsar/PulsarJsonTableSink.java | 61
.../connectors/pulsar/PulsarProduceMode.java | 37 +++
.../connectors/pulsar/PulsarTableSink.java | 163 +++
.../pulsar/partitioner/PulsarKeyExtractor.java | 36 +++
.../pulsar/serde/JsonRowDeserializationSchema.java | 130 +
.../pulsar/serde/JsonRowSerializationSchema.java | 92 ++
9 files changed, 973 insertions(+)
diff --git a/pom.xml b/pom.xml
index 4bb3cf6..1ed2802 100644
--- a/pom.xml
+++ b/pom.xml
@@ -92,6 +92,7 @@ flexible messaging model and an intuitive client
API.
pulsar-proxy
pulsar-discovery-service
pulsar-storm
+pulsar-flink
pulsar-spark
pulsar-zookeeper-utils
pulsar-testclient
@@ -167,6 +168,8 @@ flexible messaging model and an intuitive client
API.
1.8.2
2.1.1
0.206
+1.6.0
+2.11
1.15.1
diff --git a/pulsar-flink/pom.xml b/pulsar-flink/pom.xml
new file mode 100644
index 000..97ab4fb
--- /dev/null
+++ b/pulsar-flink/pom.xml
@@ -0,0 +1,137 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";
+ xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>
+ 4.0.0
+
+
+org.apache.pulsar
+pulsar
+2.2.0-incubating-SNAPSHOT
+..
+
+
+ pulsar-flink
+ Pulsar Flink Connectors
+
+
+
+
+ org.apache.flink
+ flink-streaming-java_${scala.binary.version}
+ ${flink.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-table_${scala.binary.version}
+ ${flink.version}
+ provided
+
+ true
+
+
+
+ ${project.groupId}
+ pulsar-client
+ ${project.version}
+
+
+
+ org.apache.flink
+ flink-runtime_${scala.binary.version}
+ ${flink.version}
+ test-jar
+ test
+
+
+
+ org.apache.flink
+ flink-tests_${scala.binary.version}
+ ${flink.version}
+ test
+ test-jar
+
+
+
+ org.apache.flink
+ flink-streaming-java_${scala.binary.version}
+ ${flink.version}
+ test-jar
+ test
+
+
+
+ org.apache.flink
+ flink-test-utils_${scala.binary.version}
+ ${flink.version}
+ test
+
+
+
+ org.javassist
+ javassist
+ 3.20.0-GA
+ test
+
+
+
+
+
+
+
+src/main/resources
+true
+
+
+
+
+org.apache.maven.plugins
+maven-shade-plugin
+
+
+package
+
+ shade
+
+
+ true
+
true
+
+
+ com.google.guava:guava
+
+
+
+
+ com.google
+ pulsar-flink-shade.com.google
+
+
+
+
+
+
+
+
+
diff --git
a/pulsar-flink/src/main/java/org/apache/flink/streaming/connectors/pulsar/FlinkPulsarProducer.java
b/pulsar-flink/src/main/java/org/apache/flink/streaming/connectors/pulsar/FlinkPulsarProducer.java
new file mode 100644
index 000..bddfee4
--- /dev/null
+++
b/pulsar-flink/src/main/java/org/apache/flink/streaming/connectors/pulsar/FlinkPulsarProducer.java
@@ -0,0 +1,314 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in
[GitHub] aahmed-se commented on issue #2504: [WIP] Add Presto Sql Test
aahmed-se commented on issue #2504: [WIP] Add Presto Sql Test URL: https://github.com/apache/incubator-pulsar/pull/2504#issuecomment-418472398 We haven't determined on how to proceed yet , @jerrypeng just needs this to get started. I will make this WIP for now This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #2487: Ensure standalone service comes back quickly after ungraceful restarts
merlimat commented on issue #2487: Ensure standalone service comes back quickly after ungraceful restarts URL: https://github.com/apache/incubator-pulsar/pull/2487#issuecomment-418471238 @rdhabalia I believe that having z-node created and deleted in very different parts of the code makes it very hard to figure out how things work. Having the entire logic for the lock and its cleanup within the same class encapsulate the complexity there and anyone reading that class can easily understand the behavior. Regarding `NoopLoadManager`: the standalone currently works with `ModularLoadManager`, but `ModularLoadManager` was really not designed for standalone. In fact, standalone mode doesn't need a load manager because there are no other brokers. All the features provided by a load manager are not used by standalone (and sometimes they complicate it): load-balancing, getting load reports, tracking rates of bundles, traffic shedding and so on. All this is, in the best case, it's overhead to the functioning of Pulsar standalone. The `NoopLoadManager` short circuits all these features. The only piece of code it contains is the z-node registration, since it's needed in lookup logic. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2504: Add Presto Sql Test
sijie commented on issue #2504: Add Presto Sql Test URL: https://github.com/apache/incubator-pulsar/pull/2504#issuecomment-418470600 Please update the caption and description. This doesn't actually do any real tests on the SQL execution, it is just a sanity check if the presto worker is setup correct, which doesn't match the caption and description. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2480: Allow to configure bookie settings when running in standalone mode
sijie commented on issue #2480: Allow to configure bookie settings when running in standalone mode URL: https://github.com/apache/incubator-pulsar/pull/2480#issuecomment-418467525 cherry-picked as f5a01f7 in branch-2.1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch branch-2.1 updated: Allow to configure bookie settings when running in standalone mode (#2480)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/branch-2.1 by this push: new f5a01f7 Allow to configure bookie settings when running in standalone mode (#2480) f5a01f7 is described below commit f5a01f7c82569d3a4dcfc4bed2cc94c790909117 Author: Matteo Merli AuthorDate: Tue Sep 4 11:16:11 2018 -0700 Allow to configure bookie settings when running in standalone mode (#2480) ### Motivation Most bookie configs are set in stone when running Pulsar in standalone mode. We need to be able to tune these settings to have the Pulsar standalone to adapt for running with small memory settings. --- conf/standalone.conf | 79 ++ .../java/org/apache/pulsar/PulsarStandalone.java | 9 ++- .../pulsar/zookeeper/LocalBookkeeperEnsemble.java | 20 ++ 3 files changed, 92 insertions(+), 16 deletions(-) diff --git a/conf/standalone.conf b/conf/standalone.conf index a9fb288..74a5702 100644 --- a/conf/standalone.conf +++ b/conf/standalone.conf @@ -411,3 +411,82 @@ exposePublisherStats=true # Deprecated. Use configurationStoreServers globalZookeeperServers= + +### --- BookKeeper Configuration --- # + +ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage + +# Size of Write Cache. Memory is allocated from JVM direct memory. +# Write cache is used to buffer entries before flushing into the entry log +# For good performance, it should be big enough to hold a sub +dbStorage_writeCacheMaxSizeMb=256 + +# Size of Read cache. Memory is allocated from JVM direct memory. +# This read cache is pre-filled doing read-ahead whenever a cache miss happens +dbStorage_readAheadCacheMaxSizeMb=64 + +# How many entries to pre-fill in cache after a read cache miss +dbStorage_readAheadCacheBatchSize=1000 + +flushInterval=6 + +## RocksDB specific configurations +## DbLedgerStorage uses RocksDB to store the indexes from +## (ledgerId, entryId) -> (entryLog, offset) + +# Size of RocksDB block-cache. For best performance, this cache +# should be big enough to hold a significant portion of the index +# database which can reach ~2GB in some cases +# Default is 16 MBytes +dbStorage_rocksDB_blockCacheSize=16777216 + +# Other RocksDB specific tunables +dbStorage_rocksDB_writeBufferSizeMB=4 +dbStorage_rocksDB_sstSizeInMB=4 +dbStorage_rocksDB_blockSize=4096 +dbStorage_rocksDB_bloomFilterBitsPerKey=10 +dbStorage_rocksDB_numLevels=-1 +dbStorage_rocksDB_numFilesInLevel0=4 +dbStorage_rocksDB_maxSizeInLevel1MB=256 + +# Maximum latency to impose on a journal write to achieve grouping +journalMaxGroupWaitMSec=1 + +# Should the data be fsynced on journal before acknowledgment. +journalSyncData=false + + +# For each ledger dir, maximum disk space which can be used. +# Default is 0.95f. i.e. 95% of disk can be used at most after which nothing will +# be written to that partition. If all ledger dir partions are full, then bookie +# will turn to readonly mode if 'readOnlyModeEnabled=true' is set, else it will +# shutdown. +# Valid values should be in between 0 and 1 (exclusive). +diskUsageThreshold=0.99 + +# The disk free space low water mark threshold. +# Disk is considered full when usage threshold is exceeded. +# Disk returns back to non-full state when usage is below low water mark threshold. +# This prevents it from going back and forth between these states frequently +# when concurrent writes and compaction are happening. This also prevent bookie from +# switching frequently between read-only and read-writes states in the same cases. +diskUsageWarnThreshold=0.99 + +# Whether the bookie allowed to use a loopback interface as its primary +# interface(i.e. the interface it uses to establish its identity)? +# By default, loopback interfaces are not allowed as the primary +# interface. +# Using a loopback interface as the primary interface usually indicates +# a configuration error. For example, its fairly common in some VPS setups +# to not configure a hostname, or to have the hostname resolve to +# 127.0.0.1. If this is the case, then all bookies in the cluster will +# establish their identities as 127.0.0.1:3181, and only one will be able +# to join the cluster. For VPSs configured like this, you should explicitly +# set the listening interface. +allowLoopback=true + +# How long the interval to trigger next garbage collection, in milliseconds +# Since garbage collection is running in background, too frequent gc +# will heart performance. It is better to give a higher number of gc +# interval if there is enough disk capacity. +gcWaitTime=30 diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java index b6105c9..bb30bfa 100644 --- a/pulsar-broker/src/main/java
[incubator-pulsar] branch master updated: Allow to configure bookie settings when running in standalone mode (#2480)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 8f4f371 Allow to configure bookie settings when running in standalone mode (#2480) 8f4f371 is described below commit 8f4f371def2983f64e5dfa8f31e9fe04a2e21462 Author: Matteo Merli AuthorDate: Tue Sep 4 11:16:11 2018 -0700 Allow to configure bookie settings when running in standalone mode (#2480) ### Motivation Most bookie configs are set in stone when running Pulsar in standalone mode. We need to be able to tune these settings to have the Pulsar standalone to adapt for running with small memory settings. --- conf/standalone.conf | 79 ++ .../java/org/apache/pulsar/PulsarStandalone.java | 9 ++- .../pulsar/zookeeper/LocalBookkeeperEnsemble.java | 20 ++ 3 files changed, 92 insertions(+), 16 deletions(-) diff --git a/conf/standalone.conf b/conf/standalone.conf index 09d369c..b09fcd6 100644 --- a/conf/standalone.conf +++ b/conf/standalone.conf @@ -420,3 +420,82 @@ exposePublisherStats=true # Deprecated. Use configurationStoreServers globalZookeeperServers= + +### --- BookKeeper Configuration --- # + +ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage + +# Size of Write Cache. Memory is allocated from JVM direct memory. +# Write cache is used to buffer entries before flushing into the entry log +# For good performance, it should be big enough to hold a sub +dbStorage_writeCacheMaxSizeMb=256 + +# Size of Read cache. Memory is allocated from JVM direct memory. +# This read cache is pre-filled doing read-ahead whenever a cache miss happens +dbStorage_readAheadCacheMaxSizeMb=64 + +# How many entries to pre-fill in cache after a read cache miss +dbStorage_readAheadCacheBatchSize=1000 + +flushInterval=6 + +## RocksDB specific configurations +## DbLedgerStorage uses RocksDB to store the indexes from +## (ledgerId, entryId) -> (entryLog, offset) + +# Size of RocksDB block-cache. For best performance, this cache +# should be big enough to hold a significant portion of the index +# database which can reach ~2GB in some cases +# Default is 16 MBytes +dbStorage_rocksDB_blockCacheSize=16777216 + +# Other RocksDB specific tunables +dbStorage_rocksDB_writeBufferSizeMB=4 +dbStorage_rocksDB_sstSizeInMB=4 +dbStorage_rocksDB_blockSize=4096 +dbStorage_rocksDB_bloomFilterBitsPerKey=10 +dbStorage_rocksDB_numLevels=-1 +dbStorage_rocksDB_numFilesInLevel0=4 +dbStorage_rocksDB_maxSizeInLevel1MB=256 + +# Maximum latency to impose on a journal write to achieve grouping +journalMaxGroupWaitMSec=1 + +# Should the data be fsynced on journal before acknowledgment. +journalSyncData=false + + +# For each ledger dir, maximum disk space which can be used. +# Default is 0.95f. i.e. 95% of disk can be used at most after which nothing will +# be written to that partition. If all ledger dir partions are full, then bookie +# will turn to readonly mode if 'readOnlyModeEnabled=true' is set, else it will +# shutdown. +# Valid values should be in between 0 and 1 (exclusive). +diskUsageThreshold=0.99 + +# The disk free space low water mark threshold. +# Disk is considered full when usage threshold is exceeded. +# Disk returns back to non-full state when usage is below low water mark threshold. +# This prevents it from going back and forth between these states frequently +# when concurrent writes and compaction are happening. This also prevent bookie from +# switching frequently between read-only and read-writes states in the same cases. +diskUsageWarnThreshold=0.99 + +# Whether the bookie allowed to use a loopback interface as its primary +# interface(i.e. the interface it uses to establish its identity)? +# By default, loopback interfaces are not allowed as the primary +# interface. +# Using a loopback interface as the primary interface usually indicates +# a configuration error. For example, its fairly common in some VPS setups +# to not configure a hostname, or to have the hostname resolve to +# 127.0.0.1. If this is the case, then all bookies in the cluster will +# establish their identities as 127.0.0.1:3181, and only one will be able +# to join the cluster. For VPSs configured like this, you should explicitly +# set the listening interface. +allowLoopback=true + +# How long the interval to trigger next garbage collection, in milliseconds +# Since garbage collection is running in background, too frequent gc +# will heart performance. It is better to give a higher number of gc +# interval if there is enough disk capacity. +gcWaitTime=30 diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java index b6105c9..bb30bfa 100644 --- a/pulsar-broker/src/main/java/org/apa
[GitHub] sijie closed pull request #2480: Allow to configure bookie settings when running in standalone mode
sijie closed pull request #2480: Allow to configure bookie settings when running in standalone mode URL: https://github.com/apache/incubator-pulsar/pull/2480 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/conf/standalone.conf b/conf/standalone.conf index 09d369c6ff..b09fcd6fd1 100644 --- a/conf/standalone.conf +++ b/conf/standalone.conf @@ -420,3 +420,82 @@ exposePublisherStats=true # Deprecated. Use configurationStoreServers globalZookeeperServers= + +### --- BookKeeper Configuration --- # + +ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage + +# Size of Write Cache. Memory is allocated from JVM direct memory. +# Write cache is used to buffer entries before flushing into the entry log +# For good performance, it should be big enough to hold a sub +dbStorage_writeCacheMaxSizeMb=256 + +# Size of Read cache. Memory is allocated from JVM direct memory. +# This read cache is pre-filled doing read-ahead whenever a cache miss happens +dbStorage_readAheadCacheMaxSizeMb=64 + +# How many entries to pre-fill in cache after a read cache miss +dbStorage_readAheadCacheBatchSize=1000 + +flushInterval=6 + +## RocksDB specific configurations +## DbLedgerStorage uses RocksDB to store the indexes from +## (ledgerId, entryId) -> (entryLog, offset) + +# Size of RocksDB block-cache. For best performance, this cache +# should be big enough to hold a significant portion of the index +# database which can reach ~2GB in some cases +# Default is 16 MBytes +dbStorage_rocksDB_blockCacheSize=16777216 + +# Other RocksDB specific tunables +dbStorage_rocksDB_writeBufferSizeMB=4 +dbStorage_rocksDB_sstSizeInMB=4 +dbStorage_rocksDB_blockSize=4096 +dbStorage_rocksDB_bloomFilterBitsPerKey=10 +dbStorage_rocksDB_numLevels=-1 +dbStorage_rocksDB_numFilesInLevel0=4 +dbStorage_rocksDB_maxSizeInLevel1MB=256 + +# Maximum latency to impose on a journal write to achieve grouping +journalMaxGroupWaitMSec=1 + +# Should the data be fsynced on journal before acknowledgment. +journalSyncData=false + + +# For each ledger dir, maximum disk space which can be used. +# Default is 0.95f. i.e. 95% of disk can be used at most after which nothing will +# be written to that partition. If all ledger dir partions are full, then bookie +# will turn to readonly mode if 'readOnlyModeEnabled=true' is set, else it will +# shutdown. +# Valid values should be in between 0 and 1 (exclusive). +diskUsageThreshold=0.99 + +# The disk free space low water mark threshold. +# Disk is considered full when usage threshold is exceeded. +# Disk returns back to non-full state when usage is below low water mark threshold. +# This prevents it from going back and forth between these states frequently +# when concurrent writes and compaction are happening. This also prevent bookie from +# switching frequently between read-only and read-writes states in the same cases. +diskUsageWarnThreshold=0.99 + +# Whether the bookie allowed to use a loopback interface as its primary +# interface(i.e. the interface it uses to establish its identity)? +# By default, loopback interfaces are not allowed as the primary +# interface. +# Using a loopback interface as the primary interface usually indicates +# a configuration error. For example, its fairly common in some VPS setups +# to not configure a hostname, or to have the hostname resolve to +# 127.0.0.1. If this is the case, then all bookies in the cluster will +# establish their identities as 127.0.0.1:3181, and only one will be able +# to join the cluster. For VPSs configured like this, you should explicitly +# set the listening interface. +allowLoopback=true + +# How long the interval to trigger next garbage collection, in milliseconds +# Since garbage collection is running in background, too frequent gc +# will heart performance. It is better to give a higher number of gc +# interval if there is enough disk capacity. +gcWaitTime=30 diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java index cf07d8c8ba..0e2043d1a9 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java @@ -21,6 +21,9 @@ import com.beust.jcommander.Parameter; import com.ea.agentloader.AgentLoader; import com.google.common.collect.Sets; + +import org.apache.bookkeeper.conf.ServerConfiguration; +import org.apache.commons.configuration.PropertiesConfiguration; import org.apache.pulsar.broker.PulsarService; import org.apache.pulsar.broker.ServiceConfiguration; import org.apache.pulsar.broker.ServiceConfigurationUtils; @@ -36,6 +39,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import j
[incubator-pulsar] branch branch-2.1 updated: Fix log4j config to not redirect BK logs into separate file (#2482)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new a4b20bb Fix log4j config to not redirect BK logs into separate file
(#2482)
a4b20bb is described below
commit a4b20bb28fb77a5f84be61481e126a698788ae79
Author: Matteo Merli
AuthorDate: Tue Sep 4 11:15:24 2018 -0700
Fix log4j config to not redirect BK logs into separate file (#2482)
### Motivation
Currently all `org.apache.bookkeeper` messages printed by broker are being
routed to a different log file `broker.log.bk`. Removing for now the special
config for BK logs.
---
conf/log4j2.yaml | 50 --
1 file changed, 50 deletions(-)
diff --git a/conf/log4j2.yaml b/conf/log4j2.yaml
index ab2b9ab..45ed5d9 100644
--- a/conf/log4j2.yaml
+++ b/conf/log4j2.yaml
@@ -35,10 +35,6 @@ Configuration:
value: "info"
- name: "pulsar.routing.appender.default"
value: "Console"
- - name: "bk.log.level"
-value: "info"
- - name: "bk.log.appender"
-value: "BkRollingFile"
# Example: logger-filter script
Scripts:
@@ -85,34 +81,6 @@ Configuration:
IfLastModified:
age: 30d
-# Rolling file appender configuration for bk
-RollingRandomAccessFile:
- name: BkRollingFile
- fileName: "${sys:pulsar.log.dir}/${sys:pulsar.log.file}.bk"
- filePattern:
"${sys:pulsar.log.dir}/${sys:pulsar.log.file}.bk-%d{MM-dd-}-%i.log.gz"
- immediateFlush: true
- PatternLayout:
-Pattern: "%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"
- Policies:
-TimeBasedTriggeringPolicy:
- interval: 1
- modulate: true
-SizeBasedTriggeringPolicy:
- size: 1 GB
-# Trigger every day at midnight that also scan
-# roll-over strategy that deletes older file
-CronTriggeringPolicy:
- schedule: "0 0 0 * * ?"
- # Delete file older than 30days
- DefaultRolloverStrategy:
- Delete:
-basePath: ${sys:pulsar.log.dir}
-maxDepth: 2
-IfFileName:
- glob: "*/${sys:pulsar.log.file}.bk*log.gz"
-IfLastModified:
- age: 30d
-
# Routing
Routing:
name: RoutingAppender
@@ -173,24 +141,6 @@ Configuration:
AppenderRef:
- ref: Console
- - name: org.apache.bookkeeper
-level: "${sys:bk.log.level}"
-additivity: false
-AppenderRef:
- - ref: "${sys:bk.log.appender}"
-
- - name: org.apache.distributedlog
-level: "${sys:bk.log.level}"
-additivity: false
-AppenderRef:
- - ref: "${sys:bk.log.appender}"
-
- - name: org.apache.zookeeper
-level: "${sys:bk.log.level}"
-additivity: false
-AppenderRef:
- - ref: "${sys:bk.log.appender}"
-
- name: verbose
level: info
additivity: false
[incubator-pulsar] branch master updated: Fix log4j config to not redirect BK logs into separate file (#2482)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 37f266b Fix log4j config to not redirect BK logs into separate file
(#2482)
37f266b is described below
commit 37f266bb0357fa825f1b3c2e99582526c43bba84
Author: Matteo Merli
AuthorDate: Tue Sep 4 11:15:24 2018 -0700
Fix log4j config to not redirect BK logs into separate file (#2482)
### Motivation
Currently all `org.apache.bookkeeper` messages printed by broker are being
routed to a different log file `broker.log.bk`. Removing for now the special
config for BK logs.
---
conf/log4j2.yaml | 50 --
1 file changed, 50 deletions(-)
diff --git a/conf/log4j2.yaml b/conf/log4j2.yaml
index ab2b9ab..45ed5d9 100644
--- a/conf/log4j2.yaml
+++ b/conf/log4j2.yaml
@@ -35,10 +35,6 @@ Configuration:
value: "info"
- name: "pulsar.routing.appender.default"
value: "Console"
- - name: "bk.log.level"
-value: "info"
- - name: "bk.log.appender"
-value: "BkRollingFile"
# Example: logger-filter script
Scripts:
@@ -85,34 +81,6 @@ Configuration:
IfLastModified:
age: 30d
-# Rolling file appender configuration for bk
-RollingRandomAccessFile:
- name: BkRollingFile
- fileName: "${sys:pulsar.log.dir}/${sys:pulsar.log.file}.bk"
- filePattern:
"${sys:pulsar.log.dir}/${sys:pulsar.log.file}.bk-%d{MM-dd-}-%i.log.gz"
- immediateFlush: true
- PatternLayout:
-Pattern: "%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"
- Policies:
-TimeBasedTriggeringPolicy:
- interval: 1
- modulate: true
-SizeBasedTriggeringPolicy:
- size: 1 GB
-# Trigger every day at midnight that also scan
-# roll-over strategy that deletes older file
-CronTriggeringPolicy:
- schedule: "0 0 0 * * ?"
- # Delete file older than 30days
- DefaultRolloverStrategy:
- Delete:
-basePath: ${sys:pulsar.log.dir}
-maxDepth: 2
-IfFileName:
- glob: "*/${sys:pulsar.log.file}.bk*log.gz"
-IfLastModified:
- age: 30d
-
# Routing
Routing:
name: RoutingAppender
@@ -173,24 +141,6 @@ Configuration:
AppenderRef:
- ref: Console
- - name: org.apache.bookkeeper
-level: "${sys:bk.log.level}"
-additivity: false
-AppenderRef:
- - ref: "${sys:bk.log.appender}"
-
- - name: org.apache.distributedlog
-level: "${sys:bk.log.level}"
-additivity: false
-AppenderRef:
- - ref: "${sys:bk.log.appender}"
-
- - name: org.apache.zookeeper
-level: "${sys:bk.log.level}"
-additivity: false
-AppenderRef:
- - ref: "${sys:bk.log.appender}"
-
- name: verbose
level: info
additivity: false
[GitHub] sijie closed pull request #2482: Fix log4j conf
sijie closed pull request #2482: Fix log4j conf
URL: https://github.com/apache/incubator-pulsar/pull/2482
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/conf/log4j2.yaml b/conf/log4j2.yaml
index ab2b9abc0f..45ed5d9940 100644
--- a/conf/log4j2.yaml
+++ b/conf/log4j2.yaml
@@ -35,10 +35,6 @@ Configuration:
value: "info"
- name: "pulsar.routing.appender.default"
value: "Console"
- - name: "bk.log.level"
-value: "info"
- - name: "bk.log.appender"
-value: "BkRollingFile"
# Example: logger-filter script
Scripts:
@@ -85,34 +81,6 @@ Configuration:
IfLastModified:
age: 30d
-# Rolling file appender configuration for bk
-RollingRandomAccessFile:
- name: BkRollingFile
- fileName: "${sys:pulsar.log.dir}/${sys:pulsar.log.file}.bk"
- filePattern:
"${sys:pulsar.log.dir}/${sys:pulsar.log.file}.bk-%d{MM-dd-}-%i.log.gz"
- immediateFlush: true
- PatternLayout:
-Pattern: "%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"
- Policies:
-TimeBasedTriggeringPolicy:
- interval: 1
- modulate: true
-SizeBasedTriggeringPolicy:
- size: 1 GB
-# Trigger every day at midnight that also scan
-# roll-over strategy that deletes older file
-CronTriggeringPolicy:
- schedule: "0 0 0 * * ?"
- # Delete file older than 30days
- DefaultRolloverStrategy:
- Delete:
-basePath: ${sys:pulsar.log.dir}
-maxDepth: 2
-IfFileName:
- glob: "*/${sys:pulsar.log.file}.bk*log.gz"
-IfLastModified:
- age: 30d
-
# Routing
Routing:
name: RoutingAppender
@@ -173,24 +141,6 @@ Configuration:
AppenderRef:
- ref: Console
- - name: org.apache.bookkeeper
-level: "${sys:bk.log.level}"
-additivity: false
-AppenderRef:
- - ref: "${sys:bk.log.appender}"
-
- - name: org.apache.distributedlog
-level: "${sys:bk.log.level}"
-additivity: false
-AppenderRef:
- - ref: "${sys:bk.log.appender}"
-
- - name: org.apache.zookeeper
-level: "${sys:bk.log.level}"
-additivity: false
-AppenderRef:
- - ref: "${sys:bk.log.appender}"
-
- name: verbose
level: info
additivity: false
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] rdhabalia commented on issue #2487: Ensure standalone service comes back quickly after ungraceful restarts
rdhabalia commented on issue #2487: Ensure standalone service comes back quickly after ungraceful restarts URL: https://github.com/apache/incubator-pulsar/pull/2487#issuecomment-418271067 @merlimat I still don't feel this is a good solution to address this problem because of two reasons. ModularLoadManager can support standalone broker so, no need to add another LoadManager for standalone and also we use standalone mode to test multiple broker node on local-host testing. So, I feel `PulsarStandalone` could be the right place to add `delete znode` logic rather introducing seprate load-manager. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2487: Ensure standalone service comes back quickly after ungraceful restarts
sijie commented on issue #2487: Ensure standalone service comes back quickly after ungraceful restarts URL: https://github.com/apache/incubator-pulsar/pull/2487#issuecomment-418266534 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2482: Fix log4j conf
sijie commented on issue #2482: Fix log4j conf URL: https://github.com/apache/incubator-pulsar/pull/2482#issuecomment-418266432 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2473: Downgrading ZK to stable version 3.4.13
sijie commented on issue #2473: Downgrading ZK to stable version 3.4.13 URL: https://github.com/apache/incubator-pulsar/pull/2473#issuecomment-418266324 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2480: Allow to configure bookie settings when running in standalone mode
sijie commented on issue #2480: Allow to configure bookie settings when running in standalone mode URL: https://github.com/apache/incubator-pulsar/pull/2480#issuecomment-418266191 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2488: [functions] Don't set the state storage url if state is not enabled.
sijie commented on issue #2488: [functions] Don't set the state storage url if state is not enabled. URL: https://github.com/apache/incubator-pulsar/pull/2488#issuecomment-418266116 cherry-picked 04c5c48 in branch-2.1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch branch-2.1 updated: [functions] Don't set the state storage url if state is not enabled. (#2488)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 04c5c48 [functions] Don't set the state storage url if state is not
enabled. (#2488)
04c5c48 is described below
commit 04c5c48f2fcba7724892f66acf7784518877ba3c
Author: Sijie Guo
AuthorDate: Tue Sep 4 00:17:29 2018 -0700
[functions] Don't set the state storage url if state is not enabled. (#2488)
---
pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index cf07d8c..b6105c9 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -273,7 +273,10 @@ public class PulsarStandalone implements AutoCloseable {
// worker talks to local broker
workerConfig.setPulsarServiceUrl("pulsar://127.0.0.1:" +
config.getBrokerServicePort());
workerConfig.setPulsarWebServiceUrl("http://127.0.0.1:"; +
config.getWebServicePort());
-workerConfig.setStateStorageServiceUrl("bk://127.0.0.1:" +
this.getStreamStoragePort());
+if (!this.isNoStreamStorage()) {
+// only set the state storage service url when state is
enabled.
+workerConfig.setStateStorageServiceUrl("bk://127.0.0.1:" +
this.getStreamStoragePort());
+}
String hostname =
ServiceConfigurationUtils.getDefaultOrConfiguredAddress(
config.getAdvertisedAddress());
workerConfig.setWorkerHostname(hostname);
[GitHub] sijie closed pull request #2488: [functions] Don't set the state storage url if state is not enabled.
sijie closed pull request #2488: [functions] Don't set the state storage url if
state is not enabled.
URL: https://github.com/apache/incubator-pulsar/pull/2488
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index cf07d8c8ba..b6105c9948 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -273,7 +273,10 @@ void start() throws Exception {
// worker talks to local broker
workerConfig.setPulsarServiceUrl("pulsar://127.0.0.1:" +
config.getBrokerServicePort());
workerConfig.setPulsarWebServiceUrl("http://127.0.0.1:"; +
config.getWebServicePort());
-workerConfig.setStateStorageServiceUrl("bk://127.0.0.1:" +
this.getStreamStoragePort());
+if (!this.isNoStreamStorage()) {
+// only set the state storage service url when state is
enabled.
+workerConfig.setStateStorageServiceUrl("bk://127.0.0.1:" +
this.getStreamStoragePort());
+}
String hostname =
ServiceConfigurationUtils.getDefaultOrConfiguredAddress(
config.getAdvertisedAddress());
workerConfig.setWorkerHostname(hostname);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-pulsar] branch master updated: [functions] Don't set the state storage url if state is not enabled. (#2488)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 4124de1 [functions] Don't set the state storage url if state is not
enabled. (#2488)
4124de1 is described below
commit 4124de11c407e7a5efa8aa2f4b669e22543ee427
Author: Sijie Guo
AuthorDate: Tue Sep 4 00:17:29 2018 -0700
[functions] Don't set the state storage url if state is not enabled. (#2488)
---
pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index cf07d8c..b6105c9 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -273,7 +273,10 @@ public class PulsarStandalone implements AutoCloseable {
// worker talks to local broker
workerConfig.setPulsarServiceUrl("pulsar://127.0.0.1:" +
config.getBrokerServicePort());
workerConfig.setPulsarWebServiceUrl("http://127.0.0.1:"; +
config.getWebServicePort());
-workerConfig.setStateStorageServiceUrl("bk://127.0.0.1:" +
this.getStreamStoragePort());
+if (!this.isNoStreamStorage()) {
+// only set the state storage service url when state is
enabled.
+workerConfig.setStateStorageServiceUrl("bk://127.0.0.1:" +
this.getStreamStoragePort());
+}
String hostname =
ServiceConfigurationUtils.getDefaultOrConfiguredAddress(
config.getAdvertisedAddress());
workerConfig.setWorkerHostname(hostname);
[GitHub] sijie commented on issue #2400: PIP-22: Dead Letter Topic
sijie commented on issue #2400: PIP-22: Dead Letter Topic URL: https://github.com/apache/incubator-pulsar/pull/2400#issuecomment-418265592 Closed this PR and use #2508 for it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #2400: PIP-22: Dead Letter Topic
sijie closed pull request #2400: PIP-22: Dead Letter Topic
URL: https://github.com/apache/incubator-pulsar/pull/2400
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
index 584a376fac..1d171f217e 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
@@ -23,6 +23,8 @@
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.ExecutionException;
+
import org.apache.bookkeeper.mledger.AsyncCallbacks.ClearBacklogCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.DeleteCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.FindEntryCallback;
@@ -30,6 +32,7 @@
import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.SkipEntriesCallback;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
/**
* A ManangedCursor is a persisted cursor inside a ManagedLedger.
@@ -75,6 +78,16 @@
*/
Map getProperties();
+/**
+ * Return entry at the position.
+ */
+Entry readEntry(PositionImpl position) throws InterruptedException,
ExecutionException;
+
+/**
+ * Return entry at the position async.
+ */
+void asyncReadEntry(PositionImpl position, ReadEntryCallback callback,
Object ctx);
+
/**
* Read entries from the ManagedLedger, up to the specified number. The
returned list can be smaller.
*
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
index bab354762a..7cc8aff172 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
@@ -44,7 +44,9 @@
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
@@ -385,6 +387,27 @@ public void operationFailed(ManagedLedgerException
exception) {
});
}
+@Override
+public Entry readEntry(PositionImpl position) throws InterruptedException,
ExecutionException {
+final CompletableFuture readFuture = new CompletableFuture<>();
+ledger.asyncReadEntry(position, new ReadEntryCallback() {
+@Override
+public void readEntryComplete(Entry entry, Object ctx) {
+readFuture.complete(entry);
+}
+@Override
+public void readEntryFailed(ManagedLedgerException exception,
Object ctx) {
+readFuture.completeExceptionally(exception);
+}
+}, null);
+return readFuture.get();
+}
+
+@Override
+public void asyncReadEntry(PositionImpl position, ReadEntryCallback
callback, Object ctx) {
+ledger.asyncReadEntry(position, callback, ctx);
+}
+
@Override
public List readEntries(int numberOfEntriesToRead) throws
InterruptedException, ManagedLedgerException {
checkArgument(numberOfEntriesToRead > 0);
diff --git
a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainerTest.java
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainerTest.java
index ecf6acfc50..694ffc317c 100644
---
a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainerTest.java
+++
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainerTest.java
@@ -27,6 +27,8 @@
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.ExecutionException;
+
import org.apache.bookkeeper.mledger.AsyncCallbacks;
import org.apache.bookkeeper.mledger.AsyncCallbacks.ClearBacklogCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.DeleteCallback;
@@ -60,6 +62,17 @@ public MockManagedCursor(ManagedCursorContainer container,
String name, Position
return Collections.emptyMap();
}
+@Override
+public Entry readEntry(PositionImpl position) throws
InterruptedException, ExecutionException {
+retur
[GitHub] sijie commented on a change in pull request #2504: Add Presto Sql Test
sijie commented on a change in pull request #2504: Add Presto Sql Test URL: https://github.com/apache/incubator-pulsar/pull/2504#discussion_r214807167 ## File path: tests/integration/pom.xml ## @@ -88,6 +88,12 @@ jackson-dataformat-yaml test + Review comment: define this in root pom file This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #2504: Add Presto Sql Test
sijie commented on a change in pull request #2504: Add Presto Sql Test
URL: https://github.com/apache/incubator-pulsar/pull/2504#discussion_r214807332
##
File path:
tests/integration/src/test/java/org/apache/pulsar/tests/integration/presto/TestBasicPresto.java
##
@@ -0,0 +1,47 @@
+package org.apache.pulsar.tests.integration.presto;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.tests.integration.docker.ContainerExecResult;
+import org.apache.pulsar.tests.integration.topologies.PulsarCluster;
+import org.apache.pulsar.tests.integration.topologies.PulsarClusterTestBase;
+import org.testng.ITest;
+import org.testng.annotations.AfterSuite;
+import org.testng.annotations.BeforeSuite;
+import org.testng.annotations.Test;
+
+import java.util.concurrent.TimeUnit;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+@Slf4j
+public class TestBasicPresto extends PulsarClusterTestBase implements ITest {
+
+@BeforeSuite
+public void setupCluster() throws Exception {
+super.setupCluster();
+ContainerExecResult containerExecResult =
pulsarCluster.getBroker(0).execCmd(PulsarCluster.PULSAR_COMMAND_SCRIPT,
"sql-worker", "start");
+log.info("SQL worker startup output {}",
containerExecResult.getStdout());
+assertThat(containerExecResult.getExitCode()).isEqualTo(0);
+log.info("Sleep for sql worker initialization");
+TimeUnit.MINUTES.sleep(1);
Review comment:
explain why do we need this? or can we provide a more deterministic way to
check whether sql worker is ready to be used? e.g. check http endpoint or tcp
port
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sijie commented on issue #2471: PIP-23: Pulsar Java Client Interceptors.
sijie commented on issue #2471: PIP-23: Pulsar Java Client Interceptors. URL: https://github.com/apache/incubator-pulsar/pull/2471#issuecomment-418264034 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
