[GitHub] [pulsar] codelipenghui opened a new issue #6854: Consumer stuck with batch receive when batchingMaxMessages greater than receiverQueueSize
codelipenghui opened a new issue #6854:
URL: https://github.com/apache/pulsar/issues/6854
**Describe the bug**
Consumer stuck with batch receive when batchingMaxMessages greater than
receiverQueueSize
**To Reproduce**
```java
consumer = clientProviderForTests.getPulsarClient().newConsumer()
//.subscriptionType(SubscriptionType.Shared)
.subscriptionName("disconnections3")
.topic("public/default/test")
.receiverQueueSize(500)
.batchReceivePolicy(BatchReceivePolicy.builder().maxNumBytes(1024*1024).maxNumMessages(3000).build())
.subscribe();
```
**Expected behavior**
The consumer can consume messages when `batchingMaxMessages` greater than
`receiverQueueSize`.
**Additional context**
version 2.5.1 and build from the master branch
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] jiazhai commented on pull request #6853: Add Tls with keystore type config support
jiazhai commented on pull request #6853: URL: https://github.com/apache/pulsar/pull/6853#issuecomment-622183172 @rdhabalia Thanks for the comments. The name of the PR seems be a little misleading. Changed the name to align with the main changes. It mainly want to support keystore configs. and make multi ca easy config is one result of it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] vzhikserg commented on pull request #6848: [JDBC Sink] JDBC sink does not handle null in schema
vzhikserg commented on pull request #6848: URL: https://github.com/apache/pulsar/pull/6848#issuecomment-622123559 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] rdhabalia edited a comment on pull request #6853: Add Tls with keystore type config to support multi CAs
rdhabalia edited a comment on pull request #6853: URL: https://github.com/apache/pulsar/pull/6853#issuecomment-622010332 I haven't gone through with the entire change but it seems it touches many different files and I think supporting multiple CA trust stores can be done in an easier way and might not need this big change. we can always bundle multiple CA trust store in one file and it can be provided to pulsar-broker. We also have similar kind of requirement and we are going to bundle multiple certs into one to support multiple ca usecase. https://alvinalexander.com/java/java-using-keytool-import-certificate-keystore/ We can also add script in the repo for reference which can allow user to bundle multiple certs. @merlimat can you also please review this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] rdhabalia commented on pull request #6853: Add Tls with keystore type config to support multi CAs
rdhabalia commented on pull request #6853: URL: https://github.com/apache/pulsar/pull/6853#issuecomment-622010332 I haven't gone through with the entire change but it seems it touches many different files and I think supporting multiple CA trust stores can be done in an easier way and might not need this big change. we can always bundle multiple CA trust store in one file and it can be provided to pulsar-broker. We also have similar kind of requirement and we are going to bundle multiple certs into one to support multiple ca usecase. https://alvinalexander.com/java/java-using-keytool-import-certificate-keystore/ @merlimat can you also please review this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gaoran10 edited a comment on issue #6852: [Pulsar SQL] multiple zk servers issue
gaoran10 edited a comment on issue #6852: URL: https://github.com/apache/pulsar/issues/6852#issuecomment-621885024 @KannarFr @codelipenghui Hi, currently, the method `getZkServers()` of the class `org.apache.bookkeeper.conf.ClientConfiguration` is not suited for Pulsar SQL's config component. I'll fix it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] jiazhai opened a new pull request #6853: Add Tls with keystore type config to support multi CAs
jiazhai opened a new pull request #6853: URL: https://github.com/apache/pulsar/pull/6853 Fixes #6640 ### Motivation Add Tls with keystore type config to support multi CAs ### Modifications Add Tls with keystore type config to support multi CA ### Verifying this change - Unit test passed *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API: (yes / no) - The schema: (yes / no / don't know) - The default values of configurations: (yes / no) - The wire protocol: (yes / no) - The rest endpoints: (yes / no) - The admin cli options: (yes / no) - Anything that affects deployment: (yes / no / don't know) ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on issue #6837: Can't query in S3 tired storage by using pulsar SQL(presto)
gaoran10 commented on issue #6837: URL: https://github.com/apache/pulsar/issues/6837#issuecomment-621887325 OK, I'll have a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on issue #6852: [Pulsar SQL] multiple zk servers issue
gaoran10 commented on issue #6852: URL: https://github.com/apache/pulsar/issues/6852#issuecomment-621885024 OK, I'll fix it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6837: Can't query in S3 tired storage by using pulsar SQL(presto)
codelipenghui commented on issue #6837: URL: https://github.com/apache/pulsar/issues/6837#issuecomment-621884283 @gaoran10 Could you please help take a look at this issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6824: support async send msg can get sequenceId when throw exception
codelipenghui commented on issue #6824: URL: https://github.com/apache/pulsar/issues/6824#issuecomment-621842927 close via #6825 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [Pulsar IO][Issue 5633]Support avro schema for debezium connector (#6034)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new e08be96 [Pulsar IO][Issue 5633]Support avro schema for debezium
connector (#6034)
e08be96 is described below
commit e08be96d98c7730a01be296aae38e39020ff485e
Author: guangning
AuthorDate: Thu Apr 30 21:29:05 2020 +0800
[Pulsar IO][Issue 5633]Support avro schema for debezium connector (#6034)
Fixes #5633
### Motivation
Currently, some users want to support Avro schema in debezium, so this pr
supports this feature.
For Kafka's Avro schema, it depends on the Avro 1.8 version, but Avro
version has just been upgraded to 1.9 in pulsar, so shade is needed to avoid
finding `addProp` function
### Modifications
* Add a package `kafka-connect-avro-converter-shaded`
* Add class KafkaSchema to converter Kafka's Avro schema to pulsar's schema
### Verifying this change
Unit test and integration tests
---
kafka-connect-avro-converter-shaded/pom.xml| 128 +
pom.xml| 10 ++
.../pulsar/client/impl/schema/StructSchema.java| 6 +
.../impl/schema/generic/GenericAvroReader.java | 18 ++-
.../impl/schema/generic/GenericAvroSchema.java | 4 +
.../impl/schema/generic/GenericAvroReaderTest.java | 24 +++-
.../org/apache/pulsar/functions/api/KVRecord.java | 35 +
.../org/apache/pulsar/functions/api/Record.java| 5 +
.../pulsar/functions/instance/SinkRecord.java | 25
.../apache/pulsar/functions/sink/PulsarSink.java | 38 +++--
.../org/apache/pulsar/functions/LocalRunner.java | 2 +-
pulsar-io/kafka-connect-adaptor/pom.xml| 16 +++
.../io/kafka/connect/KafkaConnectSource.java | 109 +-
.../connect/schema/KafkaSchemaWrappedSchema.java | 69 +
site2/docs/io-debezium-source.md | 18 +++
.../integration/containers/ChaosContainer.java | 1 -
.../integration/functions/PulsarFunctionsTest.java | 156 -
.../io/DebeziumMongoDbSourceTester.java| 1 +
.../integration/io/DebeziumMySqlSourceTester.java | 6 +-
.../io/DebeziumPostgreSqlSourceTester.java | 1 +
.../pulsar/tests/integration/io/SourceTester.java | 64 -
21 files changed, 645 insertions(+), 91 deletions(-)
diff --git a/kafka-connect-avro-converter-shaded/pom.xml
b/kafka-connect-avro-converter-shaded/pom.xml
new file mode 100644
index 000..1006367
--- /dev/null
+++ b/kafka-connect-avro-converter-shaded/pom.xml
@@ -0,0 +1,128 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+pulsar
+org.apache.pulsar
+2.6.0-SNAPSHOT
+..
+
+
+kafka-connect-avro-converter-shaded
+Apache Pulsar :: Kafka Connect Avro Converter shaded
+
+
+
+
+io.confluent
+kafka-connect-avro-converter
+${confluent.version}
+
+
+org.codehaus.jackson
+jackson-core-asl
+${kafka-avro-convert-jackson.version}
+
+
+org.codehaus.jackson
+jackson-mapper-asl
+${kafka-avro-convert-jackson.version}
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-shade-plugin
+
+
+package
+
+shade
+
+
+
+
+
true
+
true
+
+
+
+io.confluent:*
+
io.confluent:kafka-avro-serializer
+
io.confluent:kafka-schema-registry-client
+io.confluent:common-config
+io.confluent:common-utils
+org.apache.avro:*
+
+
org.codehaus.jackson:jackson-core-asl
+
org.codehaus.jackson:jackson-mapper-asl
+
com.thoughtworks.paranamer:paranamer
+org.xerial.snappy:snappy-java
+
org.apache.commons:commons-compress
+org.tukaani:xz
+
+
+
+
+io.confluent
+
org.apache.pulsar.kafka.shade.io.confluent
+
+
[pulsar] branch master updated: Fix Get schema by version can get the deleted schema info #6754 (#6764)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 466b0b8 Fix Get schema by version can get the deleted schema info
#6754 (#6764)
466b0b8 is described below
commit 466b0b89af61a0d9c7aa7e6220c91f366f2d3831
Author: feynmanlin <315157...@qq.com>
AuthorDate: Thu Apr 30 21:27:38 2020 +0800
Fix Get schema by version can get the deleted schema info #6754 (#6764)
---
.../service/schema/BookkeeperSchemaStorage.java| 13 +++
.../service/schema/SchemaRegistryServiceImpl.java | 37 ++-
.../broker/service/PersistentTopicE2ETest.java | 42 ++
.../broker/service/schema/SchemaServiceTest.java | 4 +--
4 files changed, 80 insertions(+), 16 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java
index df41e71..b013c8b 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java
@@ -237,10 +237,10 @@ public class BookkeeperSchemaStorage implements
SchemaStorage {
return findSchemaEntryByVersion(schemaLocator.getIndexList(),
version)
.thenApply(entry ->
-new StoredSchema(
-entry.getSchemaData().toByteArray(),
-new LongSchemaVersion(version)
-)
+new StoredSchema(
+entry.getSchemaData().toByteArray(),
+new LongSchemaVersion(version)
+)
);
});
}
@@ -387,11 +387,12 @@ public class BookkeeperSchemaStorage implements
SchemaStorage {
.setPosition(position)
.setHash(copyFrom(hash))
.build();
+
return updateSchemaLocator(getSchemaPath(schemaId),
SchemaStorageFormat.SchemaLocator.newBuilder()
.setInfo(info)
.addAllIndex(
-concat(locator.getIndexList(), newArrayList(info))
+concat(locator.getIndexList(), newArrayList(info))
).build(), locatorEntry.zkZnodeVersion
).thenApply(ignore -> nextVersion);
}
@@ -409,7 +410,7 @@ public class BookkeeperSchemaStorage implements
SchemaStorage {
SchemaStorageFormat.IndexEntry lowest = index.get(0);
if (version < lowest.getVersion()) {
return readSchemaEntry(lowest.getPosition())
-.thenCompose(entry ->
findSchemaEntryByVersion(entry.getIndexList(), version));
+.thenCompose(entry ->
findSchemaEntryByVersion(entry.getIndexList(), version));
}
for (SchemaStorageFormat.IndexEntry entry : index) {
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java
index 89eb330..827f0d8 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java
@@ -44,12 +44,14 @@ import java.util.stream.Collectors;
import javax.validation.constraints.NotNull;
import org.apache.bookkeeper.common.concurrent.FutureUtils;
+import org.apache.commons.collections.CollectionUtils;
import
org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException;
import org.apache.pulsar.broker.service.schema.proto.SchemaRegistryFormat;
import org.apache.pulsar.common.policies.data.SchemaCompatibilityStrategy;
import org.apache.pulsar.common.protocol.schema.SchemaData;
import org.apache.pulsar.common.protocol.schema.SchemaHash;
import org.apache.pulsar.common.protocol.schema.SchemaStorage;
+import org.apache.pulsar.common.protocol.schema.StoredSchema;
import org.apache.pulsar.common.schema.SchemaType;
import org.apache.pulsar.common.protocol.schema.SchemaVersion;
import org.apache.pulsar.common.util.FutureUtil;
@@ -87,15 +89,34 @@ public class SchemaRegistryServiceImpl implements
SchemaRegistryService {
@Override
@NotNull
public CompletableFuture getSchema(String schemaId,
SchemaVersion version) {
-return schemaStorage.get(schemaId, version).thenCompose(stored -> {
-if (isNull(stored)) {
-return completedFuture(null);
-} else {
-return Functions.bytesToSchemaInfo(stored.data)
-
[pulsar] branch master updated: support async send msg return sequenceId when throw Exception (#6825)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 42ed9bf support async send msg return sequenceId when throw
Exception (#6825)
42ed9bf is described below
commit 42ed9bf12e29099e5c1655484311441402dde3d7
Author: liudezhi <33149602+liudezhi2...@users.noreply.github.com>
AuthorDate: Thu Apr 30 21:26:54 2020 +0800
support async send msg return sequenceId when throw Exception (#6825)
Master Issue: #6824
## Motivation
When sending messages asynchronously fails, an exception will be thrown,
but it is not known which message is abnormal, and the user does not know which
messages need to be retried。
## Modifications
This change can be supported on the client side, when throwing an
exception need to set sequenceId
org.apache.pulsar.client.api.PulsarClientException
```java
public class PulsarClientException extends IOException {
private long sequenceId = -1;
public PulsarClientException(String msg, long sequenceId) {
super(msg);
this.sequenceId = sequenceId;
}
```
Client examples
```java
producer.newMessage().sequenceId(1).value(value.getBytes())
.sendAsync().thenAccept(msgId -> {
System.out.println(msgId);
}).exceptionally(ex -> {
System.out.println(
((PulsarClientException)ex.getCause()).getSequenceId());
return null;
});
```
---
.../client/api/SimpleProducerConsumerTest.java | 39 +++
.../pulsar/client/api/PulsarClientException.java | 121 -
.../apache/pulsar/client/impl/ProducerImpl.java| 33 +++---
3 files changed, 177 insertions(+), 16 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
index 51a7eef..a2fde6c 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
@@ -47,6 +47,7 @@ import java.nio.file.Paths;
import java.time.Clock;
import java.time.Instant;
import java.time.ZoneId;
+import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
@@ -1143,6 +1144,44 @@ public class SimpleProducerConsumerTest extends
ProducerConsumerBase {
}
@Test
+public void testtSendCallBackReturnSequenceId() throws Exception {
+log.info("-- Starting {} test --", methodName);
+
+ProducerBuilder producerBuilder = pulsarClient.newProducer()
+.enableBatching(false)
+.topic("persistent://my-property/my-ns/my-topic5")
+.sendTimeout(1, TimeUnit.SECONDS);
+
+Producer producer = producerBuilder.create();
+final String message = "my-message";
+
+// Trigger the send timeout
+stopBroker();
+List> futures = new
ArrayList>();
+for(int i = 0 ; i < 3 ; i++) {
+ CompletableFuture future =
producer.newMessage().sequenceId(i).value(message.getBytes()).sendAsync();
+ futures.add(future);
+}
+Thread.sleep(3000);
+futures.get(0).exceptionally(ex -> {
+long sequenceId = ((PulsarClientException)
ex.getCause()).getSequenceId();
+Assert.assertEquals(sequenceId, 0L);
+return null;
+});
+futures.get(1).exceptionally(ex -> {
+long sequenceId = ((PulsarClientException)
ex.getCause()).getSequenceId();
+Assert.assertEquals(sequenceId, 1L);
+return null;
+});
+futures.get(2).exceptionally(ex -> {
+long sequenceId = ((PulsarClientException)
ex.getCause()).getSequenceId();
+Assert.assertEquals(sequenceId, 2L);
+return null;
+});
+log.info("-- Exiting {} test --", methodName);
+}
+
+@Test
public void testSendCallBack() throws Exception {
log.info("-- Starting {} test --", methodName);
diff --git
a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/PulsarClientException.java
b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/PulsarClientException.java
index 607c3db..16af009 100644
---
a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/PulsarClientException.java
+++
b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/PulsarClientException.java
@@ -27,7 +27,7 @@ import java.util.concurrent.ExecutionException;
*/
@SuppressWarnings("serial")
public class PulsarClientException extends IOException {
-
+
[pulsar] branch master updated: Add null check for offload policy(#6775)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 0951828 Add null check for offload policy(#6775)
0951828 is described below
commit 0951828d66430b587fdb9e5642a8fb24a106e17a
Author: luceneReader <541313...@qq.com>
AuthorDate: Thu Apr 30 21:23:54 2020 +0800
Add null check for offload policy(#6775)
---
.../java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
index bb584a7..5de3915 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
@@ -2557,8 +2557,9 @@ public abstract class NamespacesBase extends
AdminResource {
byte[] content = globalZk().getData(path, null, nodeStat);
Policies policies = jsonMapper().readValue(content,
Policies.class);
-if (offloadPolicies.getManagedLedgerOffloadDeletionLagInMillis()
-
.equals(OffloadPolicies.DEFAULT_OFFLOAD_DELETION_LAG_IN_MILLIS)) {
+if (offloadPolicies.getManagedLedgerOffloadDeletionLagInMillis()
== null && OffloadPolicies.DEFAULT_OFFLOAD_DELETION_LAG_IN_MILLIS == null
+||
offloadPolicies.getManagedLedgerOffloadDeletionLagInMillis() != null
+&&
offloadPolicies.getManagedLedgerOffloadDeletionLagInMillis().equals(OffloadPolicies.DEFAULT_OFFLOAD_DELETION_LAG_IN_MILLIS))
{
offloadPolicies.setManagedLedgerOffloadDeletionLagInMillis(policies.offload_deletion_lag_ms);
} else {
policies.offload_deletion_lag_ms =
offloadPolicies.getManagedLedgerOffloadDeletionLagInMillis();
[GitHub] [pulsar] complex64 commented on issue #5847: Error at starting apachepulsar/pulsar-dashboard docker container within Kubernetes
complex64 commented on issue #5847: URL: https://github.com/apache/pulsar/issues/5847#issuecomment-621828763 > You can try http://pulsar.apache.org/docs/en/administration-pulsar-manager/ first. https://github.com/apache/pulsar-manager is a better management tool. Let's take a look at this problem next. @complex64 Thanks for the pointer to the manager, we'll give that project a try :+1: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui edited a comment on issue #6852: [Pulsar SQL] multiple zk servers issue
codelipenghui edited a comment on issue #6852: URL: https://github.com/apache/pulsar/issues/6852#issuecomment-621822923 @gaoran10 could you please help take a look at this issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6852: [Pulsar SQL] multiple zk servers issue
codelipenghui commented on issue #6852: URL: https://github.com/apache/pulsar/issues/6852#issuecomment-621822923 @gaoran10 could you please help take a look this issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] KannarFr opened a new issue #6852: [Pulsar SQL] multiple zk servers issue
KannarFr opened a new issue #6852: URL: https://github.com/apache/pulsar/issues/6852 **Describe the bug** Pulsar SQL documentation indicates: ``` pulsar.zookeeper-uri=localhost1,localhost2:2181 ``` It doesn't work, I tried: ``` pulsar.zookeeper-uri=localhost1:2181,localhost2:2181 ``` And same issue. But for ``` pulsar.zookeeper-uri=localhost1:2181 ``` It works. When it doesn't work, it throws: ``` 2020-04-30T12:33:53.229ZINFO Query-20200430_123351_0_ak294-159 org.apache.zookeeper.common.X509Util Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2020-04-30T12:33:53.233ZINFO Query-20200430_123351_0_ak294-159 org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2020-04-30T12:33:53.239ZINFO Query-20200430_123351_0_ak294-159 org.apache.zookeeper.ClientCnxn zookeeper.request.timeout value is 0. feature enabled= 2020-04-30T12:33:53.247ZINFO Query-20200430_123351_0_ak294-159-SendThread(yo-zookeeper-c2-n3:2184) org.apache.zookeeper.ClientCnxnOpening socket connection to server yo-zookeeper-c2-n3/192.168.10.6:2184. Will not attempt to authenticate using SASL (unknown error) 2020-04-30T12:33:53.253ZINFO Query-20200430_123351_0_ak294-159-SendThread(yo-zookeeper-c2-n3:2184) org.apache.zookeeper.ClientCnxnSocket connection established, initiating session, client: /192.168.10.17:56234, server: yo-zookeeper-c2-n3/192.168.10.6:2184 2020-04-30T12:33:53.273ZINFO Query-20200430_123351_0_ak294-159-SendThread(yo-zookeeper-c2-n3:2184) org.apache.zookeeper.ClientCnxnSession establishment complete on server yo-zookeeper-c2-n3/192.168.10.6:2184, sessionid = 0x308edaa37a20019, negotiated timeout = 1 2020-04-30T12:33:53.276ZINFO Query-20200430_123351_0_ak294-159-EventThread org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBaseZooKeeper client is connected now. 2020-04-30T12:33:53.306ZINFO Query-20200430_123351_0_ak294-159 org.apache.bookkeeper.meta.MetadataDrivers BookKeeper metadata driver manager initialized 2020-04-30T12:33:53.307ZINFO Query-20200430_123351_0_ak294-159 org.apache.bookkeeper.meta.MetadataDrivers BookKeeper metadata driver manager initialized 2020-04-30T12:33:53.307ZINFO Query-20200430_123351_0_ak294-159 org.apache.bookkeeper.meta.MetadataDrivers BookKeeper metadata driver manager initialized 2020-04-30T12:33:53.310ZINFO Query-20200430_123351_0_ak294-159 org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase Initialize zookeeper metadata driver with external zookeeper client : ledgersRootPath = /ledgers. 2020-04-30T12:33:53.324ZWARN Query-20200430_123351_0_ak294-159 org.apache.bookkeeper.util.EventLoopUtilCould not use Netty Epoll event loop: failed to load the required native library 2020-04-30T12:33:53.347ZERROR Query-20200430_123351_0_ak294-159 org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl Failed to initialize DNS Resolver org.apache.bookkeeper.net.ScriptBasedMapping, used default subnet resolver : java.lang.RuntimeException: No network topology script is found when using script based DNS resolver. 2020-04-30T12:33:53.353ZINFO Query-20200430_123351_0_ak294-159 org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl Initialize rackaware ensemble placement policy @ @ /default-rack : org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$DefaultResolver. 2020-04-30T12:33:53.353ZINFO Query-20200430_123351_0_ak294-159 org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl Not weighted 2020-04-30T12:33:53.356ZINFO Query-20200430_123351_0_ak294-159 org.apache.bookkeeper.client.BookKeeper Weighted ledger placement is not enabled 2020-04-30T12:33:53.383ZINFO BookKeeperClientScheduler-OrderedScheduler-0-0 org.apache.bookkeeper.net.NetworkTopologyImpl Adding a new node: /default-rack/yo-bookkeeper-c1-n3:3181 2020-04-30T12:33:53.383ZINFO BookKeeperClientScheduler-OrderedScheduler-0-0 org.apache.bookkeeper.net.NetworkTopologyImpl Adding a new node: /default-rack/yo-bookkeeper-c1-n1:3181 2020-04-30T12:33:53.384ZINFO BookKeeperClientScheduler-OrderedScheduler-0-0 org.apache.bookkeeper.net.NetworkTopologyImpl Adding a new node: /default-rack/yo-bookkeeper-c1-n2:3181 2020-04-30T12:33:53.386ZERROR Query-20200430_123351_0_ak294-159 org.apache.pulsar.sql.presto.PulsarSplitManager Failed to get splits java.io.IOException: Failed to initialize ledger manager factory at
[GitHub] [pulsar] massakam opened a new pull request #6851: [broker] Enable users to set subscription expiration time for each namespace
massakam opened a new pull request #6851: URL: https://github.com/apache/pulsar/pull/6851 ### Motivation We can automatically delete inactive subscriptions by setting `subscriptionExpirationTimeMinutes` in broker.conf to a value greater than 0. ```sh # How long to delete inactive subscriptions from last consuming # When it is 0, inactive subscriptions are not deleted automatically subscriptionExpirationTimeMinutes=0 ``` However, since this setting value applies to all topics, we have to set it to 0 if there is even one topic whose subscriptions should not be deleted. ### Modifications Enable users to set a subscription expiration time for each namespace. This value overrides `subscriptionExpirationTimeMinutes` in broker.conf. ```sh $ ./bin/pulsar-admin namespaces set-subscription-expiration-time --time 60 tenant1/ns1 $ ./bin/pulsar-admin namespaces get-subscription-expiration-time tenant1/ns1 60 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] itaross commented on issue #6845: Subscription to topics with regex not working for non-persisten topics
itaross commented on issue #6845: URL: https://github.com/apache/pulsar/issues/6845#issuecomment-621711182 I am currently using version 2.5.1 I think you can reproduce it even with a builting connector. Create it from the coommand line, using --topics-pattern and use a non-persistent:// based pattern. You will see that no data is received in the write method. This has been highlighted also in the previous cited issued, but it has not been addressed in sinks. Everything works if you you use persistent topics. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] jiazhai commented on issue #6834: Apache pulsar Encryption failed using ecdsa
jiazhai commented on issue #6834: URL: https://github.com/apache/pulsar/issues/6834#issuecomment-621700480 Cpp and python client is not involved in the FIPS provider support. We may need to confirm this feature is supported, and provide an example. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Persi edited a comment on issue #6842: Pulsar Java client contains shaded Jackson which makes it kind of unusable with Spring Boot
Persi edited a comment on issue #6842: URL: https://github.com/apache/pulsar/issues/6842#issuecomment-621651699 **UPDATE:** Using `pulsar-client-original` does not work either... seems like the ObjectMapper instance of JSONSchema does not load/use available Jackson extensions. Which brings me back to my second point (which actually don't even has to do with Spring Boot). As there is currently no way to configure or provide an ObjectMapper instance, it is impossible to reuse already configured ObjectMapper instances. I would suggest to add a provider interface for ObjectMapper with the current implementation from JSONSchema as default implementation. That would allow to provide other ObjectMapper instances independent of the environment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] mileschao commented on pull request #237: [Issue #200] feature: add internal connectionReader readAtLeast error information
mileschao commented on pull request #237: URL: https://github.com/apache/pulsar-client-go/pull/237#issuecomment-621662203 @sijie @wolfstudy I've add some error information for #200 It help to solve problem. :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #6827: [Issue 6822][C++] Fix segmentation fault if handleSendTimeout() is called after closeAsync()
BewareMyPower commented on pull request #6827: URL: https://github.com/apache/pulsar/pull/6827#issuecomment-621656984 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] mileschao opened a new pull request #237: [Issue #200] feature: add internal connectionReader readAtLeast error information
mileschao opened a new pull request #237: URL: https://github.com/apache/pulsar-client-go/pull/237 Master Issue: #200 ### Motivation add internal connectionReader readAtLeast error information these error information may help to solve #200 ### Modifications function ```readAtLeast``` return error type ### Verifying this change This change is already covered by existing tests, such as *(please describe tests)*. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (yes / ***no***) - The public API: (yes / ***no***) - The schema: (yes / ***no*** / don't know) - The default values of configurations: (yes / ***no***) - The wire protocol: (yes / ***no***) ### Documentation - Does this pull request introduce a new feature? (yes / ***no***) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Persi commented on issue #6842: Pulsar Java client contains shaded Jackson which makes it kind of unusable with Spring Boot
Persi commented on issue #6842: URL: https://github.com/apache/pulsar/issues/6842#issuecomment-621651699 Using `pulsar-client-original` seems to work as expected :) But my second point (which actually don't even has to do with Spring Boot) is already an issue. As there is currently no way to configure or provide an ObjectMapper instance, it is impossible to reuse already configured ObjectMapper instances. I would suggest to add a provider interface for ObjectMapper with the current implementation from JSONSchema as default implementation. That would allow to provide other ObjectMapper instances independent of the environment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6845: Subscription to topics with regex not working for non-persisten topics
codelipenghui commented on issue #6845: URL: https://github.com/apache/pulsar/issues/6845#issuecomment-621642156 @itaross Can you provide more details about the bug? what version are you using and how to reproduce it. This will help to troubleshoot this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
