[GitHub] [pulsar] eolivelli commented on pull request #10595: [Pulsar IO] Develop Huawei Cloud DIS connector
eolivelli commented on pull request #10595: URL: https://github.com/apache/pulsar/pull/10595#issuecomment-841604321 Can you please explain better this problem with 'merge policy' ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits
eolivelli commented on pull request #7266: URL: https://github.com/apache/pulsar/pull/7266#issuecomment-841603849 @merlimat do you want to take a look as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] yangl opened a new pull request #10597: Fix pulsar-client & pulsar-client-admin shaded jar missing code source
yangl opened a new pull request #10597: URL: https://github.com/apache/pulsar/pull/10597 ### Motivation when we use the `pulsar-client` and the `pulsar-client-admin` dependency in the ide(such as the IntelliJ IDEA), which cannot link the code source correctly because of the both jar was shaded by the maven-shade-plugin.   ### Modifications 1. add the `true` configuration in the [maven-shade-plugin.](https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#createSourcesJar) 2. change the maven-source-plugin execution phase to `package` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #9925: [pulsar-broker] Namespace-resource use namespace name to fetch policy resource
codelipenghui commented on pull request #9925: URL: https://github.com/apache/pulsar/pull/9925#issuecomment-841585101 Hi Rajan @rdhabalia, Could you please help resolve the conflicts? so that we can merge it before 2.8.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Shoothzj commented on pull request #10595: [Pulsar IO] Develop Huawei Cloud DIS connector
Shoothzj commented on pull request #10595: URL: https://github.com/apache/pulsar/pull/10595#issuecomment-841585081 For the merge policy, I close this PR, I will commit these codes to https://github.com/streamnative/pulsar-io-huawei-dis -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Shoothzj closed pull request #10595: [Pulsar IO] Develop Huawei Cloud DIS connector
Shoothzj closed pull request #10595: URL: https://github.com/apache/pulsar/pull/10595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [Issue 3127][python-client] Replace Exceptions with PulsarExceptions (#7600)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 3f36544 [Issue 3127][python-client] Replace Exceptions with PulsarExceptions (#7600) 3f36544 is described below commit 3f36544c76620a7c6cd91c8047ecc28dbc3f85db Author: Livio Benčik AuthorDate: Sat May 15 04:15:48 2021 +0200 [Issue 3127][python-client] Replace Exceptions with PulsarExceptions (#7600) Fixes #7600 ### Motivation As the issue says, the Python client throws `Exception`s instead of a subclass of it (`PulsarException`), so the clients must catch the blanket Exception. ### Modifications Every C `PulsarException` is now thrown in Python with the same type. --- pulsar-client-cpp/python/CMakeLists.txt | 3 +- pulsar-client-cpp/python/pulsar/__init__.py | 2 + pulsar-client-cpp/python/pulsar/exceptions.py | 28 + pulsar-client-cpp/python/pulsar_test.py | 171 +- pulsar-client-cpp/python/src/enums.cc | 13 ++ pulsar-client-cpp/python/src/exceptions.cc| 92 ++ pulsar-client-cpp/python/src/pulsar.cc| 6 +- 7 files changed, 201 insertions(+), 114 deletions(-) diff --git a/pulsar-client-cpp/python/CMakeLists.txt b/pulsar-client-cpp/python/CMakeLists.txt index e78d80a..c110d01 100644 --- a/pulsar-client-cpp/python/CMakeLists.txt +++ b/pulsar-client-cpp/python/CMakeLists.txt @@ -29,7 +29,8 @@ ADD_LIBRARY(_pulsar SHARED src/pulsar.cc src/authentication.cc src/reader.cc src/schema.cc - src/cryptoKeyReader.cc) + src/cryptoKeyReader.cc + src/exceptions.cc) SET(CMAKE_SHARED_LIBRARY_PREFIX ) SET(CMAKE_SHARED_LIBRARY_SUFFIX .so) diff --git a/pulsar-client-cpp/python/pulsar/__init__.py b/pulsar-client-cpp/python/pulsar/__init__.py index b47c87d..3f00554 100644 --- a/pulsar-client-cpp/python/pulsar/__init__.py +++ b/pulsar-client-cpp/python/pulsar/__init__.py @@ -103,6 +103,8 @@ import _pulsar from _pulsar import Result, CompressionType, ConsumerType, InitialPosition, PartitionsRoutingMode, BatchingType # noqa: F401 +from pulsar.exceptions import * + from pulsar.functions.function import Function from pulsar.functions.context import Context from pulsar.functions.serde import SerDe, IdentitySerDe, PickleSerDe diff --git a/pulsar-client-cpp/python/pulsar/exceptions.py b/pulsar-client-cpp/python/pulsar/exceptions.py new file mode 100644 index 000..d151564 --- /dev/null +++ b/pulsar-client-cpp/python/pulsar/exceptions.py @@ -0,0 +1,28 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +from _pulsar import PulsarException, UnknownError, InvalidConfiguration, Timeout, LookupError, ConnectError, \ +ReadError, AuthenticationError, AuthorizationError, ErrorGettingAuthenticationData, BrokerMetadataError, \ +BrokerPersistenceError, ChecksumError, ConsumerBusy, NotConnected, AlreadyClosed, InvalidMessage, \ +ConsumerNotInitialized, ProducerNotInitialized, ProducerBusy, TooManyLookupRequestException, InvalidTopicName, \ +InvalidUrl, ServiceUnitNotReady, OperationNotSupported, ProducerBlockedQuotaExceededError, \ +ProducerBlockedQuotaExceededException, ProducerQueueIsFull, MessageTooBig, TopicNotFound, SubscriptionNotFound, \ +ConsumerNotFound, UnsupportedVersionError, TopicTerminated, CryptoError, IncompatibleSchema, ConsumerAssignError, \ +CumulativeAcknowledgementNotAllowedError, TransactionCoordinatorNotFoundError, InvalidTxnStatusError, \ +NotAllowedError, TransactionConflict, TransactionNotFound, ProducerFenced, MemoryBufferIsFull diff --git a/pulsar-client-cpp/python/pulsar_test.py b/pulsar-client-cpp/python/pulsar_test.py index e7d05f3..71b67d7 100755 --- a/pulsar-client-cpp/python/pulsar_test.py +++ b/pulsar-client-cpp/python/pulsar_test.py @@ -42,6 +42,7 @@ except ImportError: TM = 1 # Do not wait forever in tests + def doHttpPost(url, data): req =
[GitHub] [pulsar] codelipenghui merged pull request #7600: [Issue 3127][python-client] Replace Exceptions with PulsarExceptions
codelipenghui merged pull request #7600: URL: https://github.com/apache/pulsar/pull/7600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Fix the default retry letter and dead letter topic name (#10129)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 86d0830 Fix the default retry letter and dead letter topic name
(#10129)
86d0830 is described below
commit 86d0830b7b173970424266fcf2f065f6df171020
Author: WangJialing <65590138+wangjialing...@users.noreply.github.com>
AuthorDate: Sat May 15 10:01:54 2021 +0800
Fix the default retry letter and dead letter topic name (#10129)
### Motivation
Fixes #9327
### Modifications
Correct the default retry letter and dead letter topic name depend on full
topic name
---
.../apache/pulsar/client/api/RetryTopicTest.java | 86 +-
.../pulsar/client/impl/ConsumerBuilderImpl.java| 24 +++---
2 files changed, 98 insertions(+), 12 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/RetryTopicTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/RetryTopicTest.java
index 3840efa..fc84a62 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/RetryTopicTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/RetryTopicTest.java
@@ -19,6 +19,7 @@
package org.apache.pulsar.client.api;
import lombok.Cleanup;
+import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testng.annotations.AfterMethod;
@@ -68,7 +69,85 @@ public class RetryTopicTest extends ProducerConsumerBase {
@Cleanup
PulsarClient newPulsarClient = newPulsarClient(lookupUrl.toString(),
0);// Creates new client connection
Consumer deadLetterConsumer =
newPulsarClient.newConsumer(Schema.BYTES)
-.topic("persistent://my-property/my-ns/my-subscription-DLQ")
+
.topic("persistent://my-property/my-ns/retry-topic-my-subscription-DLQ")
+.subscriptionName("my-subscription")
+
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+.subscribe();
+
+Producer producer = pulsarClient.newProducer(Schema.BYTES)
+.topic(topic)
+.create();
+
+for (int i = 0; i < sendMessages; i++) {
+producer.send(String.format("Hello Pulsar [%d]", i).getBytes());
+}
+
+producer.close();
+
+int totalReceived = 0;
+do {
+Message message = consumer.receive();
+log.info("consumer received message : {} {}",
message.getMessageId(), new String(message.getData()));
+consumer.reconsumeLater(message, 1 , TimeUnit.SECONDS);
+totalReceived++;
+} while (totalReceived < sendMessages * (maxRedeliveryCount + 1));
+
+int totalInDeadLetter = 0;
+do {
+Message message = deadLetterConsumer.receive();
+log.info("dead letter consumer received message : {} {}",
message.getMessageId(), new String(message.getData()));
+deadLetterConsumer.acknowledge(message);
+totalInDeadLetter++;
+} while (totalInDeadLetter < sendMessages);
+
+deadLetterConsumer.close();
+consumer.close();
+
+Consumer checkConsumer =
this.pulsarClient.newConsumer(Schema.BYTES)
+.topic(topic)
+.subscriptionName("my-subscription")
+.subscriptionType(SubscriptionType.Shared)
+
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+.subscribe();
+
+Message checkMessage = checkConsumer.receive(3,
TimeUnit.SECONDS);
+if (checkMessage != null) {
+log.info("check consumer received message : {} {}",
checkMessage.getMessageId(), new String(checkMessage.getData()));
+}
+assertNull(checkMessage);
+
+checkConsumer.close();
+}
+
+//Issue 9327: do compatibility check in case of the default retry and dead
letter topic name changed
+@Test
+public void testRetryTopicNameForCompatibility () throws Exception {
+final String topic = "persistent://my-property/my-ns/retry-topic";
+
+final String oldRetryTopic =
"persistent://my-property/my-ns/my-subscription-RETRY";
+
+final String oldDeadLetterTopic =
"persistent://my-property/my-ns/my-subscription-DLQ";
+
+final int maxRedeliveryCount = 2;
+
+final int sendMessages = 100;
+
+admin.topics().createPartitionedTopic(oldRetryTopic, 2);
+admin.topics().createPartitionedTopic(oldDeadLetterTopic, 2);
+
+Consumer consumer = pulsarClient.newConsumer(Schema.BYTES)
+.topic(topic)
+.subscriptionName("my-subscription")
+.subscriptionType(SubscriptionType.Shared)
+.enableRetry(true)
+
[GitHub] [pulsar] codelipenghui merged pull request #10129: fix the default retry letter and dead letter topic name
codelipenghui merged pull request #10129: URL: https://github.com/apache/pulsar/pull/10129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #9327: Retry and Dead letter topics are generated with wrong names when retry is enabled
codelipenghui closed issue #9327: URL: https://github.com/apache/pulsar/issues/9327 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated (96325b9 -> 7bd849b)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 96325b9 [CLI] Add support for setting time based limit on backlog quota with CLI. (#10401) add 7bd849b Reader support seek from separate messageId/time (#10348) No new revisions were added by this update. Summary of changes: .../pulsar/client/impl/MultiTopicsReaderTest.java | 93 -- .../org/apache/pulsar/client/impl/ReaderTest.java | 43 ++ .../org/apache/pulsar/client/api/Consumer.java | 13 ++- .../java/org/apache/pulsar/client/api/Reader.java | 33 .../pulsar/client/impl/MultiTopicsReaderImpl.java | 11 +++ .../org/apache/pulsar/client/impl/ReaderImpl.java | 11 +++ 6 files changed, 195 insertions(+), 9 deletions(-)
[pulsar] branch master updated (cd7e3c0 -> 96325b9)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from cd7e3c0 Make failPendingMessages called from within the ProducerImpl object mutex (#10528) add 96325b9 [CLI] Add support for setting time based limit on backlog quota with CLI. (#10401) No new revisions were added by this update. Summary of changes: conf/broker.conf | 3 +++ conf/standalone.conf | 3 +++ .../java/org/apache/pulsar/broker/admin/AdminResource.java | 7 +-- .../org/apache/pulsar/admin/cli/PulsarAdminToolTest.java | 12 ++-- .../main/java/org/apache/pulsar/admin/cli/CmdNamespaces.java | 5 - .../src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java | 5 - site2/docs/admin-api-namespaces.md | 4 ++-- site2/docs/cookbooks-retention-expiry.md | 5 +++-- 8 files changed, 34 insertions(+), 10 deletions(-)
[GitHub] [pulsar] codelipenghui merged pull request #10348: Reader support seek from separate messageId/time
codelipenghui merged pull request #10348: URL: https://github.com/apache/pulsar/pull/10348 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #9301: MultiTopicsReaderImpl – Start from separate MessageId for each topic/ partition
codelipenghui closed issue #9301: URL: https://github.com/apache/pulsar/issues/9301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #9680: Backlog quota: allow time-based quota
codelipenghui closed issue #9680: URL: https://github.com/apache/pulsar/issues/9680 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #10401: [CLI] Add support for setting time based limit on backlog quota with CLI.
codelipenghui merged pull request #10401: URL: https://github.com/apache/pulsar/pull/10401 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated (4a8d40c -> cd7e3c0)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 4a8d40c Fix partitioned system topic check bug (#10529) add cd7e3c0 Make failPendingMessages called from within the ProducerImpl object mutex (#10528) No new revisions were added by this update. Summary of changes: .../org/apache/pulsar/client/impl/ProducerImpl.java | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-)
[pulsar] branch master updated (74c0c31 -> 4a8d40c)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 74c0c31 Reduce function name size to 53 for k8s runtime (#10531) add 4a8d40c Fix partitioned system topic check bug (#10529) No new revisions were added by this update. Summary of changes: .../apache/pulsar/broker/systopic/SystemTopicClient.java | 5 + .../systopic/NamespaceEventsSystemTopicServiceTest.java| 14 ++ 2 files changed, 19 insertions(+)
[GitHub] [pulsar] codelipenghui merged pull request #10528: Make failPendingMessages called from within the ProducerImpl object mutex
codelipenghui merged pull request #10528: URL: https://github.com/apache/pulsar/pull/10528 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #10529: Fix partitioned system topic check bug
codelipenghui merged pull request #10529: URL: https://github.com/apache/pulsar/pull/10529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated (841de89 -> 74c0c31)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 841de89 [CI] Run pulsar-broker tests that don't belong to any TestNG group (#10590) add 74c0c31 Reduce function name size to 53 for k8s runtime (#10531) No new revisions were added by this update. Summary of changes: .../apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [pulsar] codelipenghui merged pull request #10531: K8s Function Name Length Check Allows Invalid StatefulSet
codelipenghui merged pull request #10531: URL: https://github.com/apache/pulsar/pull/10531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] ta1meng commented on issue #10596: Python client: broken/missing Avro schema support
ta1meng commented on issue #10596: URL: https://github.com/apache/pulsar/issues/10596#issuecomment-841574871 This ticket is a superset of https://github.com/apache/pulsar/issues/9571 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] ta1meng opened a new issue #10596: Python client: broken/missing Avro schema support
ta1meng opened a new issue #10596: URL: https://github.com/apache/pulsar/issues/10596 **Describe the bug** There are many issues in the Python client library regarding Avro schema support: * The documented [example](https://pulsar.apache.org/docs/fr/client-libraries-python/#complex-types) does not work. * No support for `default` values. * No support for the `doc` attribute. * No support for the `namespace` attribute. * No support for required subrecords (required fields of type `record`) * Ordering bug when generating schema definition that makes schema incompatible with topic's registered schema. **To Reproduce** Steps to reproduce the behavior: 1. Go to https://github.com/ta1meng/pulsar-python-avro-schema-examples 2. Read the README files 3. All issues above except one can be reproduced using the Python example applications in that repo 4. The repo does not contain an example of the lack of support of explicit default values, but it's easy to see that shortcoming in code -- there is no code for `default` values inside the schema generation code. 5. Apply the changes in the `schema_fix` directory to the Pulsar Python client library, so that all example applications will succeed. **Expected behavior** The [documented example](https://pulsar.apache.org/docs/fr/client-libraries-python/#complex-types) should work out of the box. There should be support for common Avro keywords like `default`, `doc`, and `namespace`. **Screenshots** N/A **Desktop (please complete the following information):** - OS: MacOS **Additional context** Tested agains Pulsar 2.7.1. Using Python Pulsar client 2.7.1 under Python 3.7. Though issues and fixes should be appliable to Python 2.7 as well (untested). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Shoothzj opened a new pull request #10595: [Pulsar IO] Develop Huawei Cloud DIS connector
Shoothzj opened a new pull request #10595: URL: https://github.com/apache/pulsar/pull/10595 ### Motivation We are trying to use **Pulsar IO** connector. The **DIS** is one of Sink my business must supported. ### Modifications Add a new **Pulsar IO** module DIS. ### Verifying this change - Fill the Junit test for config file. - Test the E2E ability manually. ### Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] merlimat opened a new pull request #10594: Fixed missed ZK caching when fetching list of namespaces for a tenant
merlimat opened a new pull request #10594: URL: https://github.com/apache/pulsar/pull/10594 ### Motivation The ZK reads triggered by the "get namespaces of a tenant" admin operation are being issued directly on the ZK client and therefore are not getting cached by brokers. This change is only for 2.7 branch. The issue is not present in current master since the code was already ported to use MetadataStore API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] david-streamlio opened a new pull request #10593: [Issue-10109] [admin client] Add --batch-source-config switch to the Pulsar Admin Source API
david-streamlio opened a new pull request #10593: URL: https://github.com/apache/pulsar/pull/10593 Fixes #10109 ### Motivation We are unable to create BatchSource connectors using the Pulsar Admin CLI, as there is no way to provide the batch source configuration other than providing it inside the source config yaml file. This fix provides an alternative means of providing these configuration vaules. ### Modifications Modified org.apache.pulsar.admin.cli.CmdSources and created associated unit tests ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change added tests and can be verified as follows: - Added unit tests that submit BatchSource class from the command line using the `--batch-source-config` switch ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: no) - The rest endpoints: (no) - The admin cli options: (yes) - Anything that affects deployment: (no) ### Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] rdhabalia commented on a change in pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits
rdhabalia commented on a change in pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#discussion_r632844450
##
File path: conf/bkenv.sh
##
@@ -39,7 +39,7 @@ BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms2g -Xmx2g
-XX:MaxDirectMemorySize=2g
BOOKIE_GC=${BOOKIE_GC:-${PULSAR_GC:-"-XX:+UseG1GC -XX:MaxGCPauseMillis=10
-XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions
-XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32
-XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB"}}
# Extra options to be passed to the jvm
-BOOKIE_EXTRA_OPTS="${BOOKIE_EXTRA_OPTS:-"-Dio.netty.leakDetectionLevel=disabled
${PULSAR_EXTRA_OPTS:-"-Dio.netty.recycler.maxCapacity.default=1000
-Dio.netty.recycler.linkCapacity=1024"}"} ${BOOKIE_MEM} ${BOOKIE_GC}"
+BOOKIE_EXTRA_OPTS="${BOOKIE_EXTRA_OPTS:-"-Dio.netty.leakDetectionLevel=advanced
${PULSAR_EXTRA_OPTS:-"-Dio.netty.recycler.maxCapacity.default=1000
-Dio.netty.recycler.linkCapacity=1024"}"} ${BOOKIE_MEM} ${BOOKIE_GC}"
Review comment:
oops.. good catch. I kept it for testing. fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits
eolivelli commented on a change in pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#discussion_r632765919
##
File path: conf/bkenv.sh
##
@@ -39,7 +39,7 @@ BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms2g -Xmx2g
-XX:MaxDirectMemorySize=2g
BOOKIE_GC=${BOOKIE_GC:-${PULSAR_GC:-"-XX:+UseG1GC -XX:MaxGCPauseMillis=10
-XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions
-XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32
-XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB"}}
# Extra options to be passed to the jvm
-BOOKIE_EXTRA_OPTS="${BOOKIE_EXTRA_OPTS:-"-Dio.netty.leakDetectionLevel=disabled
${PULSAR_EXTRA_OPTS:-"-Dio.netty.recycler.maxCapacity.default=1000
-Dio.netty.recycler.linkCapacity=1024"}"} ${BOOKIE_MEM} ${BOOKIE_GC}"
+BOOKIE_EXTRA_OPTS="${BOOKIE_EXTRA_OPTS:-"-Dio.netty.leakDetectionLevel=advanced
${PULSAR_EXTRA_OPTS:-"-Dio.netty.recycler.maxCapacity.default=1000
-Dio.netty.recycler.linkCapacity=1024"}"} ${BOOKIE_MEM} ${BOOKIE_GC}"
Review comment:
Are you sure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] atezs82 commented on pull request #10574: [Issue 905][api][pulsar-client] add serializable token class
atezs82 commented on pull request #10574: URL: https://github.com/apache/pulsar/pull/10574#issuecomment-841453177 @merlimat I pushed a slightly different version without new classes (just modified `AuthenticationToken`). Let me know what do you think. Note that in theory it is still possible to provide a not-serializable token supplier, which would still give back that cryptic NullPointerException above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated (55376af -> 841de89)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 55376af Fix flaky test GracefulExecutorServicesShutdownTest.shouldTerminateWhenFutureIsCancelled (#10592) add 841de89 [CI] Run pulsar-broker tests that don't belong to any TestNG group (#10590) No new revisions were added by this update. Summary of changes: build/run_unit_group.sh | 2 +- .../org/apache/pulsar/tests/AnnotationListener.java | 20 2 files changed, 21 insertions(+), 1 deletion(-)
[GitHub] [pulsar] merlimat merged pull request #10590: [CI] Run pulsar-broker tests that don't belong to any TestNG group
merlimat merged pull request #10590: URL: https://github.com/apache/pulsar/pull/10590 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] merlimat closed issue #10589: Tests that don't belong to any TestNG group aren't run at all in pulsar-broker module
merlimat closed issue #10589: URL: https://github.com/apache/pulsar/issues/10589 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Fix flaky test GracefulExecutorServicesShutdownTest.shouldTerminateWhenFutureIsCancelled (#10592)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 55376af Fix flaky test
GracefulExecutorServicesShutdownTest.shouldTerminateWhenFutureIsCancelled
(#10592)
55376af is described below
commit 55376af6cc2481c368b734c9286d05a839320db1
Author: linlinnn
AuthorDate: Sat May 15 02:17:17 2021 +0800
Fix flaky test
GracefulExecutorServicesShutdownTest.shouldTerminateWhenFutureIsCancelled
(#10592)
* fix flaky test
GracefulExecutorServicesShutdownTest.shouldTerminateWhenFutureIsCancelled
* avoid import *
---
.../pulsar/broker/service/GracefulExecutorServicesShutdownTest.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/GracefulExecutorServicesShutdownTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/GracefulExecutorServicesShutdownTest.java
index b978507..726f37d 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/GracefulExecutorServicesShutdownTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/GracefulExecutorServicesShutdownTest.java
@@ -35,6 +35,7 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
import org.testng.annotations.Test;
+import org.awaitility.Awaitility;
public class GracefulExecutorServicesShutdownTest {
@@ -151,8 +152,8 @@ public class GracefulExecutorServicesShutdownTest {
future.cancel(false);
// then
-assertTrue(awaitTerminationInterrupted.get(),
-"awaitTermination should have been interrupted");
+Awaitility.await().untilAsserted(() ->
assertTrue(awaitTerminationInterrupted.get(),
+"awaitTermination should have been interrupted"));
verify(executorService, times(1)).awaitTermination(anyLong(), any());
verify(executorService, times(1)).shutdownNow();
}
[GitHub] [pulsar] lhotari merged pull request #10592: Fix flaky test GracefulExecutorServicesShutdownTest.shouldTerminateWhenFutureIsCancelled
lhotari merged pull request #10592: URL: https://github.com/apache/pulsar/pull/10592 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] david-streamlio commented on a change in pull request #10370: [Issue-10269] : [Functions] : Pulsar IO Sink errors aren't bubbled up properly
david-streamlio commented on a change in pull request #10370:
URL: https://github.com/apache/pulsar/pull/10370#discussion_r632709623
##
File path:
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
##
@@ -317,46 +322,77 @@ private void setupStateStore() throws Exception {
}
}
-private void processResult(Record srcRecord,
- CompletableFuture result)
throws Exception {
-result.whenComplete((result1, throwable) -> {
-if (throwable != null || result1.getUserException() != null) {
-Throwable t = throwable != null ? throwable :
result1.getUserException();
-log.warn("Encountered exception when processing message {}",
-srcRecord, t);
-stats.incrUserExceptions(t);
-srcRecord.fail();
-} else {
-if (result1.getResult() != null) {
-sendOutputMessage(srcRecord, result1.getResult());
+private void processResult(@SuppressWarnings("rawtypes") Record srcRecord,
+ JavaExecutionResult result) throws
SinkException {
+
+ if (result.getUserException() != null) {
Review comment:
This is where we do the error handling for the synchronous function
calls. The logic inside this block is identical to error handling logic for
asynchronous function calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] jerrypeng commented on a change in pull request #10370: [Issue-10269] : [Functions] : Pulsar IO Sink errors aren't bubbled up properly
jerrypeng commented on a change in pull request #10370:
URL: https://github.com/apache/pulsar/pull/10370#discussion_r632667651
##
File path:
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
##
@@ -317,46 +322,77 @@ private void setupStateStore() throws Exception {
}
}
-private void processResult(Record srcRecord,
- CompletableFuture result)
throws Exception {
-result.whenComplete((result1, throwable) -> {
-if (throwable != null || result1.getUserException() != null) {
-Throwable t = throwable != null ? throwable :
result1.getUserException();
-log.warn("Encountered exception when processing message {}",
-srcRecord, t);
-stats.incrUserExceptions(t);
-srcRecord.fail();
-} else {
-if (result1.getResult() != null) {
-sendOutputMessage(srcRecord, result1.getResult());
+private void processResult(@SuppressWarnings("rawtypes") Record srcRecord,
+ JavaExecutionResult result) throws
SinkException {
+
+ if (result.getUserException() != null) {
+stats.incrUserExceptions(result.getUserException());
+srcRecord.fail();
+return;
+ }
+
+ if (result.isAsync()) {
+ result.getFuture().whenComplete((result1, throwable) -> {
+ if (throwable != null) {
+ Throwable t = throwable;
+ log.warn("Encountered exception when processing message {}",
srcRecord, t);
+ stats.incrUserExceptions(t);
+ srcRecord.fail();
+ } else {
+if (result1 != null) {
+ try {
+sendOutputMessage(srcRecord, result1);
+ } catch (SinkException e) {
+log.warn("Encountered exception when publishing result
{}", srcRecord, e);
+ }
} else {
-if (instanceConfig.getFunctionDetails().getAutoAck()) {
-// the function doesn't produce any result or the user
doesn't want the result.
-srcRecord.ack();
-}
+ if (instanceConfig.getFunctionDetails().getAutoAck()) {
+// the function doesn't produce any result or the user
doesn't want the result.
+srcRecord.ack();
+ }
}
// increment total successfully processed
stats.incrTotalProcessedSuccessfully();
+ }
+ });
+ } else {
Review comment:
Whether sync or async the error handling should be consistent:
https://github.com/apache/pulsar/pull/10370/files#diff-4b164878665f89af44f9ace0a3df8a27f4f2fb700fd9981d663fb3a7e4317e7aR338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] jerrypeng commented on a change in pull request #10370: [Issue-10269] : [Functions] : Pulsar IO Sink errors aren't bubbled up properly
jerrypeng commented on a change in pull request #10370:
URL: https://github.com/apache/pulsar/pull/10370#discussion_r632663569
##
File path:
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
##
@@ -317,46 +322,77 @@ private void setupStateStore() throws Exception {
}
}
-private void processResult(Record srcRecord,
- CompletableFuture result)
throws Exception {
-result.whenComplete((result1, throwable) -> {
-if (throwable != null || result1.getUserException() != null) {
-Throwable t = throwable != null ? throwable :
result1.getUserException();
-log.warn("Encountered exception when processing message {}",
-srcRecord, t);
-stats.incrUserExceptions(t);
-srcRecord.fail();
-} else {
-if (result1.getResult() != null) {
-sendOutputMessage(srcRecord, result1.getResult());
+private void processResult(@SuppressWarnings("rawtypes") Record srcRecord,
+ JavaExecutionResult result) throws
SinkException {
+
+ if (result.getUserException() != null) {
+stats.incrUserExceptions(result.getUserException());
+srcRecord.fail();
+return;
+ }
+
+ if (result.isAsync()) {
+ result.getFuture().whenComplete((result1, throwable) -> {
+ if (throwable != null) {
+ Throwable t = throwable;
+ log.warn("Encountered exception when processing message {}",
srcRecord, t);
+ stats.incrUserExceptions(t);
+ srcRecord.fail();
+ } else {
+if (result1 != null) {
+ try {
+sendOutputMessage(srcRecord, result1);
+ } catch (SinkException e) {
+log.warn("Encountered exception when publishing result
{}", srcRecord, e);
+ }
} else {
-if (instanceConfig.getFunctionDetails().getAutoAck()) {
-// the function doesn't produce any result or the user
doesn't want the result.
-srcRecord.ack();
-}
+ if (instanceConfig.getFunctionDetails().getAutoAck()) {
+// the function doesn't produce any result or the user
doesn't want the result.
+srcRecord.ack();
+ }
}
// increment total successfully processed
stats.incrTotalProcessedSuccessfully();
+ }
+ });
+ } else {
Review comment:
In this code path we still need to check if there was a user exception
thrown in the function and fail the record is there was.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar] branch master updated: Added more unit tests to the JavaInstanceTest class (#10369)
This is an automated email from the ASF dual-hosted git repository.
jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new c4098d6 Added more unit tests to the JavaInstanceTest class (#10369)
c4098d6 is described below
commit c4098d6845194ca78f3ddbc6748ec3c3bb313cb9
Author: David Kjerrumgaard <35466513+david-stream...@users.noreply.github.com>
AuthorDate: Fri May 14 09:47:37 2021 -0700
Added more unit tests to the JavaInstanceTest class (#10369)
---
.../functions/instance/JavaInstanceTest.java | 96 ++
1 file changed, 96 insertions(+)
diff --git
a/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/JavaInstanceTest.java
b/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/JavaInstanceTest.java
index 59f36d9..b1e7cc7 100644
---
a/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/JavaInstanceTest.java
+++
b/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/JavaInstanceTest.java
@@ -21,6 +21,8 @@ package org.apache.pulsar.functions.instance;
import static org.mockito.Mockito.mock;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertSame;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
@@ -54,6 +56,35 @@ public class JavaInstanceTest {
assertEquals(new String(testString + "-lambda"),
result.get().getResult());
instance.close();
}
+
+@Test
+public void testNullReturningFunction() throws Exception {
+ JavaInstance instance = new JavaInstance(
+mock(ContextImpl.class),
+(Function) (input, context) -> null,
+new InstanceConfig());
+ String testString = "ABC123";
+ CompletableFuture result =
instance.handleMessage(mock(Record.class), testString);
+ assertNull(result.get().getResult());
+ instance.close();
+}
+
+@Test
+public void testUserExceptionThrowingFunction() throws Exception {
+ final UserException userException = new UserException("Boom");
+ Function func = (input, context) -> {
+ throw userException;
+ };
+
+ JavaInstance instance = new JavaInstance(
+mock(ContextImpl.class),
+func,
+new InstanceConfig());
+ String testString = "ABC123";
+ CompletableFuture result =
instance.handleMessage(mock(Record.class), testString);
+ assertSame(userException, result.get().getUserException());
+ instance.close();
+}
@Test
public void testAsyncFunction() throws Exception {
@@ -86,6 +117,64 @@ public class JavaInstanceTest {
assertEquals(new String(testString + "-lambda"),
result.get().getResult());
instance.close();
}
+
+@Test
+public void testNullReturningAsyncFunction() throws Exception {
+InstanceConfig instanceConfig = new InstanceConfig();
+@Cleanup("shutdownNow")
+ExecutorService executor = Executors.newCachedThreadPool();
+
+Function> function = (input,
context) -> {
+log.info("input string: {}", input);
+CompletableFuture result = new CompletableFuture<>();
+executor.submit(() -> {
+try {
+Thread.sleep(500);
+result.complete(null);
+} catch (Exception e) {
+result.completeExceptionally(e);
+}
+});
+
+return result;
+};
+
+JavaInstance instance = new JavaInstance(
+mock(ContextImpl.class),
+function,
+instanceConfig);
+String testString = "ABC123";
+CompletableFuture result =
instance.handleMessage(mock(Record.class), testString);
+assertNull(result.get().getResult());
+instance.close();
+}
+
+@Test
+public void testUserExceptionThrowingAsyncFunction() throws Exception {
+ final UserException userException = new UserException("Boom");
+InstanceConfig instanceConfig = new InstanceConfig();
+@Cleanup("shutdownNow")
+ExecutorService executor = Executors.newCachedThreadPool();
+
+Function> function = (input,
context) -> {
+log.info("input string: {}", input);
+CompletableFuture result = new CompletableFuture<>();
+executor.submit(() -> {
+ result.completeExceptionally(userException);
+});
+
+return result;
+};
+
+JavaInstance instance = new JavaInstance(
+mock(ContextImpl.class),
+function,
+
[GitHub] [pulsar] jerrypeng merged pull request #10369: Added more unit tests to the JavaInstanceTest class
jerrypeng merged pull request #10369: URL: https://github.com/apache/pulsar/pull/10369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #10498: SinkContext: ability to seek/pause/resume consumer for a topic
dlg99 commented on a change in pull request #10498:
URL: https://github.com/apache/pulsar/pull/10498#discussion_r632659889
##
File path:
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
##
@@ -706,4 +714,67 @@ public void close() {
logger.warn("Failed to close producers", e);
}
}
+
+@Override
+public void seek(String topic, int partition, MessageId messageId) throws
PulsarClientException {
+Consumer consumer = getConsumer(topic, partition);
+consumer.seek(messageId);
+}
+
+@Override
+public void pause(String topic, int partition) throws
PulsarClientException {
+getConsumer(topic, partition).pause();
+}
+
+@Override
+public void resume(String topic, int partition) throws
PulsarClientException {
+getConsumer(topic, partition).resume();
+}
+
+public void setInputConsumers(List> inputConsumers) {
+this.inputConsumers = inputConsumers;
+inputConsumers.stream()
+.flatMap(consumer ->
+consumer instanceof MultiTopicsConsumerImpl
+? ((MultiTopicsConsumerImpl)
consumer).getConsumers().stream()
+: Stream.of(consumer))
+.forEach(consumer ->
topicConsumers.putIfAbsent(TopicName.get(consumer.getTopic()), consumer));
+}
+
+@VisibleForTesting
+Consumer getConsumer(String topic, int partition) throws
PulsarClientException {
+if (inputConsumers == null) {
+throw new PulsarClientException("Getting consumer is not
supported");
+}
+for (int i = 0; i < 2; i++) {
Review comment:
Two attempts to get the consumer:
1. Try to get the consumer.
2. If not found, reprocess MultiTopicsConsumers in case new consumers
appeared (happens on repartition or a new topic that matches provided pattern
if the pattern is used),
3. give it another try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] jerrypeng commented on a change in pull request #10498: SinkContext: ability to seek/pause/resume consumer for a topic
jerrypeng commented on a change in pull request #10498:
URL: https://github.com/apache/pulsar/pull/10498#discussion_r632656974
##
File path:
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
##
@@ -706,4 +714,67 @@ public void close() {
logger.warn("Failed to close producers", e);
}
}
+
+@Override
+public void seek(String topic, int partition, MessageId messageId) throws
PulsarClientException {
+Consumer consumer = getConsumer(topic, partition);
+consumer.seek(messageId);
+}
+
+@Override
+public void pause(String topic, int partition) throws
PulsarClientException {
+getConsumer(topic, partition).pause();
+}
+
+@Override
+public void resume(String topic, int partition) throws
PulsarClientException {
+getConsumer(topic, partition).resume();
+}
+
+public void setInputConsumers(List> inputConsumers) {
+this.inputConsumers = inputConsumers;
+inputConsumers.stream()
+.flatMap(consumer ->
+consumer instanceof MultiTopicsConsumerImpl
+? ((MultiTopicsConsumerImpl)
consumer).getConsumers().stream()
+: Stream.of(consumer))
+.forEach(consumer ->
topicConsumers.putIfAbsent(TopicName.get(consumer.getTopic()), consumer));
+}
+
+@VisibleForTesting
+Consumer getConsumer(String topic, int partition) throws
PulsarClientException {
+if (inputConsumers == null) {
+throw new PulsarClientException("Getting consumer is not
supported");
+}
+for (int i = 0; i < 2; i++) {
Review comment:
Why is this for loop needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Joshhw closed issue #7830: Pulsar Proxy doesn't forward credentials correctly with custom authN/authZ
Joshhw closed issue #7830: URL: https://github.com/apache/pulsar/issues/7830 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Joshhw commented on issue #7830: Pulsar Proxy doesn't forward credentials correctly with custom authN/authZ
Joshhw commented on issue #7830: URL: https://github.com/apache/pulsar/issues/7830#issuecomment-841298543 I'm not sure this is an issue anymore as I think the `authenticateOriginalAuthData` seemed to work out. I didn't even realize this was still open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] linlinnn opened a new pull request #10592: fix flaky test GracefulExecutorServicesShutdownTest.shouldTerminateWh…
linlinnn opened a new pull request #10592: URL: https://github.com/apache/pulsar/pull/10592 **Motivation** fix flaky test GracefulExecutorServicesShutdownTest.shouldTerminateWhenFutureIsCancelled -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] linlinnn commented on pull request #10590: [CI] Run pulsar-broker tests that don't belong to any TestNG group
linlinnn commented on pull request #10590: URL: https://github.com/apache/pulsar/pull/10590#issuecomment-841276983 There is a flaky test reported in CI. ``` Error: shouldTerminateWhenFutureIsCancelled(org.apache.pulsar.broker.service.GracefulExecutorServicesShutdownTest) Time elapsed: 0.027 s <<< FAILURE! java.lang.AssertionError: awaitTermination should have been interrupted expected [true] but found [false] at org.testng.Assert.fail(Assert.java:99) at org.testng.Assert.failNotEquals(Assert.java:1037) at org.testng.Assert.assertTrue(Assert.java:45) at org.apache.pulsar.broker.service.GracefulExecutorServicesShutdownTest.shouldTerminateWhenFutureIsCancelled(GracefulExecutorServicesShutdownTest.java:154) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #10591: Upgrade NodeJs in Pulsar Build image (used by WebSite Builder)
eolivelli commented on pull request #10591: URL: https://github.com/apache/pulsar/pull/10591#issuecomment-841268851 After building this image we have to push it do dockerhub. I do not have the privileges @merlimat -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli opened a new pull request #10591: Upgrade NodeJs in Pulsar Build image (used by WebSite Builder)
eolivelli opened a new pull request #10591: URL: https://github.com/apache/pulsar/pull/10591 The website builder CI job uses Node JS but it fails because now it requires a more recent version of Node. We also have to change the PIP 2.7 download script -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari opened a new pull request #10590: [Tests] Fix running of all pulsar-broker tests
lhotari opened a new pull request #10590: URL: https://github.com/apache/pulsar/pull/10590 Fixes #10589 ### Motivation See #10589 . Tests that don't belong to any TestNG group aren't run at all in pulsar-broker module. ### Modifications * Add logic to Pulsar's TestNG listener to add a test method to "other" group when no group has been specified. * Run the "other" group as part of broker_group_2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] 13/14: Fix expired tls certs for cpp tests (#9607)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit feb511d431997c2d821f891a81c4d45193c1bdc8
Author: Sijie Guo
AuthorDate: Wed Feb 17 15:55:57 2021 -0800
Fix expired tls certs for cpp tests (#9607)
* Fix expired tls certs for cpp tests
* Added "admin" role as super user
* Fixed common name in certs
* Added admin name in one more place
* One more place missing "admin"
* Fixed PulsarFunctionLocalRunTest
* More fixes
Co-authored-by: Matteo Merli
---
.../api/AuthenticatedProducerConsumerTest.java | 1 +
.../AuthenticationTlsHostnameVerificationTest.java | 1 +
.../worker/PulsarFunctionLocalRunTest.java | 2 +-
.../worker/PulsarFunctionPublishTest.java | 2 +-
.../functions/worker/PulsarFunctionTlsTest.java| 2 +-
.../worker/PulsarWorkerAssignmentTest.java | 2 +-
.../apache/pulsar/io/PulsarFunctionAdminTest.java | 2 +-
.../apache/pulsar/io/PulsarFunctionE2ETest.java| 2 +-
.../apache/pulsar/io/PulsarFunctionTlsTest.java| 4 +-
.../resources/authentication/tls/broker-cert.pem | 166 +
.../resources/authentication/tls/broker-key.pem| 55 ---
.../test/resources/authentication/tls/cacert.pem | 163 ++--
.../resources/authentication/tls/client-cert.pem | 147 ++
.../resources/authentication/tls/client-key.pem| 55 ---
.../tls/hn-verification/broker-cert.pem| 105 -
.../tls/hn-verification/broker-key.pem | 52 +++
.../authentication/tls/hn-verification/cacert.pem | 104 -
pulsar-client-cpp/test-conf/standalone-ssl.conf| 2 +-
18 files changed, 444 insertions(+), 423 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticatedProducerConsumerTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticatedProducerConsumerTest.java
index 08ea8c6..c544f66 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticatedProducerConsumerTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticatedProducerConsumerTest.java
@@ -82,6 +82,7 @@ public class AuthenticatedProducerConsumerTest extends
ProducerConsumerBase {
superUserRoles.add("localhost");
superUserRoles.add("superUser");
superUserRoles.add("superUser2");
+superUserRoles.add("admin");
conf.setSuperUserRoles(superUserRoles);
conf.setBrokerClientAuthenticationPlugin(AuthenticationTls.class.getName());
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticationTlsHostnameVerificationTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticationTlsHostnameVerificationTest.java
index 54a847f..27ebcaa 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticationTlsHostnameVerificationTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticationTlsHostnameVerificationTest.java
@@ -78,6 +78,7 @@ public class AuthenticationTlsHostnameVerificationTest
extends ProducerConsumerB
superUserRoles.add("localhost");
superUserRoles.add("superUser");
superUserRoles.add("superUser2");
+superUserRoles.add("admin");
conf.setSuperUserRoles(superUserRoles);
conf.setBrokerClientAuthenticationPlugin(AuthenticationTls.class.getName());
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java
index 99ff990..ebcef42 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java
@@ -147,7 +147,7 @@ public class PulsarFunctionLocalRunTest {
config = spy(new ServiceConfiguration());
config.setClusterName(CLUSTER);
-Set superUsers = Sets.newHashSet("superUser");
+Set superUsers = Sets.newHashSet("superUser", "admin");
config.setSuperUserRoles(superUsers);
config.setWebServicePort(Optional.of(0));
config.setWebServicePortTls(Optional.of(0));
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionPublishTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionPublishTest.java
index 2eca82e..187e138 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionPublishTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionPublishTest.java
@@ -131,7 +131,7 @@ public class
[pulsar] 11/14: Add AvroSchema UUID support
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit d5aeb665c9d6c11219809a21801fb575b6ee33ed
Author: Vincent Royer
AuthorDate: Thu Apr 29 00:03:47 2021 +0200
Add AvroSchema UUID support
---
.../org/apache/pulsar/client/impl/schema/AvroSchema.java | 1 +
.../apache/pulsar/client/impl/schema/util/SchemaUtil.java | 8 ++--
.../apache/pulsar/client/impl/schema/AvroSchemaTest.java | 15 +++
3 files changed, 22 insertions(+), 2 deletions(-)
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroSchema.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroSchema.java
index d5a99f8..e405497 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroSchema.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroSchema.java
@@ -122,6 +122,7 @@ public class AvroSchema extends AvroBaseStructSchema {
// Skip if have not provide joda-time dependency.
}
}
+reflectData.addLogicalTypeConversion(new Conversions.UUIDConversion());
}
}
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/util/SchemaUtil.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/util/SchemaUtil.java
index 7e1e1c0..b73fb03 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/util/SchemaUtil.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/util/SchemaUtil.java
@@ -18,10 +18,12 @@
*/
package org.apache.pulsar.client.impl.schema.util;
+import org.apache.avro.Conversions;
import org.apache.avro.Schema;
import org.apache.avro.reflect.ReflectData;
import org.apache.commons.lang3.StringUtils;
import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.client.impl.schema.AvroSchema;
import org.apache.pulsar.client.impl.schema.SchemaDefinitionBuilderImpl;
import org.apache.pulsar.common.schema.SchemaInfo;
import org.apache.pulsar.common.schema.SchemaType;
@@ -88,8 +90,10 @@ public class SchemaUtil {
try {
return
parseAvroSchema(pojo.getDeclaredField("SCHEMA$").get(null).toString());
} catch (NoSuchFieldException | IllegalAccessException |
IllegalArgumentException ignored) {
-return schemaDefinition.getAlwaysAllowNull() ?
ReflectData.AllowNull.get().getSchema(pojo)
-: ReflectData.get().getSchema(pojo);
+ReflectData reflectData = schemaDefinition.getAlwaysAllowNull() ?
ReflectData.AllowNull.get()
+: ReflectData.get();
+AvroSchema.addLogicalTypeConversions(reflectData,
schemaDefinition.isJsr310ConversionEnabled());
+return reflectData.getSchema(pojo);
}
}
}
diff --git
a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
index dcf3abe..340c53c 100644
---
a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
+++
b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
@@ -36,6 +36,8 @@ import java.util.Arrays;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
+import java.util.UUID;
+
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
@@ -426,4 +428,17 @@ public class AvroSchemaTest {
assertEquals(field1, foo.getField1());
}
+static class MyPojo {
+public UUID uid;
+}
+
+@Test
+public void testAvroUUID() {
+org.apache.pulsar.client.api.Schema schema =
org.apache.pulsar.client.api.Schema.AVRO(MyPojo.class);
+MyPojo pojo1 = new MyPojo();
+pojo1.uid = UUID.randomUUID();
+MyPojo pojo2 = schema.decode(schema.encode(pojo1));
+assertEquals(pojo1.uid, pojo2.uid);
+}
+
}
[pulsar] 10/14: [Build] Specify release in maven-compiler-plugin configuration on JDK11 (#10343)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 1a94bbcdc7b197f1167652e894de093ab7cf7be2
Author: Lari Hotari
AuthorDate: Mon Apr 26 00:56:56 2021 +0300
[Build] Specify release in maven-compiler-plugin configuration on JDK11
(#10343)
* Specify release in maven-compiler-plugin configuration
* Use "8" instead of "1.8"
(cherry picked from commit e5a2c5f272808163fd8c588e3e4badc6b5e70ca3)
---
pom.xml | 22 +-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 258fabb..1cec75a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -74,6 +74,9 @@ flexible messaging model and an intuitive client
API.
+8
+8
+
*
@@ -1505,13 +1508,30 @@ flexible messaging model and an intuitive client
API.
- jdk11-tests
+ jdk11
[11,)
+
+
+
${maven.compiler.target}
+
--add-opens
java.base/jdk.internal.loader=ALL-UNNAMED
+
+
+
+
+ maven-compiler-plugin
+
+
+${maven.compiler.release}
+
+
+
+
+
coverage
[pulsar] 14/14: [Issue 8751] Update Dockerfile for Pulsar and Dashboard to Create and Use pulsar User (nonroot user) (#8796)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 376867a1f9bd729566065e4ca13b1983c426e6f8
Author: Michael Marshall <47911938+michaeljmarsh...@users.noreply.github.com>
AuthorDate: Mon Feb 15 21:17:52 2021 -0700
[Issue 8751] Update Dockerfile for Pulsar and Dashboard to Create and Use
pulsar User (nonroot user) (#8796)
Fixes #8751
Pulsar does not need to run as the root user. This PR updates the pulsar
and the pulsar dashboard images to make them run as a new `pulsar` user (user
~1000~ 1 and group 10001). This change increases the security of pulsar
images.
Update two `Dockerfile`s to create a pulsar user, chown the appropriate
directories, and then use that user by default.
- [ ] Make sure that the change passes the CI checks.
I manually verified that the docker images run with the correct user and
file permissions. As this is my first commit, I'm not familiar with pulsar
testing. Are there tests that run against the produced docker images? If so,
then there is likely no further testing needed.
(cherry picked from commit 4264a67a84d9a9f4a49cebc591c46b252dcf4e45)
---
docker/pulsar-all/Dockerfile | 12
docker/pulsar-standalone/Dockerfile| 3 ++
docker/pulsar/Dockerfile | 33 +-
site2/docs/getting-started-docker.md | 5
.../docker-images/latest-version-image/Dockerfile | 6
.../latest-version-image/conf/bookie.conf | 1 +
.../latest-version-image/conf/broker.conf | 1 +
.../conf/functions_worker.conf | 1 +
.../latest-version-image/conf/global-zk.conf | 1 +
.../latest-version-image/conf/local-zk.conf| 1 +
.../latest-version-image/conf/presto_worker.conf | 3 +-
.../latest-version-image/conf/proxy.conf | 1 +
12 files changed, 60 insertions(+), 8 deletions(-)
diff --git a/docker/pulsar-all/Dockerfile b/docker/pulsar-all/Dockerfile
index 5daa406..8a5451d 100644
--- a/docker/pulsar-all/Dockerfile
+++ b/docker/pulsar-all/Dockerfile
@@ -25,8 +25,20 @@ ARG PULSAR_OFFLOADER_TARBALL
ADD ${PULSAR_IO_DIR} /connectors
ADD ${PULSAR_OFFLOADER_TARBALL} /
RUN mv /apache-pulsar-offloaders-*/offloaders /offloaders
+RUN chmod -R g=u /connectors /offloaders
FROM apachepulsar/pulsar:latest
+
+# Need permission to create directories and update file permissions
+USER root
+
+RUN mkdir /pulsar/connectors /pulsar/offloaders && \
+chown pulsar:root /pulsar/connectors /pulsar/offloaders && \
+chmod g=u /pulsar/connectors /pulsar/offloaders
+
+# Return to pulsar (non root) user
+USER pulsar
+
COPY --from=pulsar-all /connectors/pulsar-io-elastic-search-*.nar
/pulsar/connectors/
COPY --from=pulsar-all /connectors/pulsar-io-kinesis-*.nar /pulsar/connectors/
COPY --from=pulsar-all /connectors/pulsar-io-kafka-*.nar /pulsar/connectors/
diff --git a/docker/pulsar-standalone/Dockerfile
b/docker/pulsar-standalone/Dockerfile
index 777541b..c7fdad9 100644
--- a/docker/pulsar-standalone/Dockerfile
+++ b/docker/pulsar-standalone/Dockerfile
@@ -26,6 +26,9 @@ FROM apachepulsar/pulsar-dashboard:latest as dashboard
# Restart from
FROM openjdk:11-jdk
+# Help to make these directories persist between container restarts
+VOLUME ["/pulsar/conf", "/pulsar/data"]
+
# Note that the libpq-dev package is needed here in order to install
# the required python psycopg2 package (for postgresql) later
RUN apt-get update \
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index a224c33..0dc6598 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -17,7 +17,8 @@
# under the License.
#
-# First create a stage with just the Pulsar tarball and scripts
+# First create a stage with the Pulsar tarball, the scripts, the python client,
+# the cpp client, and the data directory. Then ensure correct file permissions.
FROM busybox as pulsar
ARG PULSAR_TARBALL
@@ -34,12 +35,25 @@ COPY scripts/watch-znode.py /pulsar/bin
COPY scripts/set_python_version.sh /pulsar/bin
COPY scripts/install-pulsar-client-37.sh /pulsar/bin
+COPY target/python-client/ /pulsar/pulsar-client
+COPY target/cpp-client/ /pulsar/cpp-client
+
+RUN mkdir /pulsar/data
+
+# In order to support running this docker image as a container on OpenShift
+# the final image needs to give the root group enough permission.
+# The file permissions are maintained when copied into the target image.
+RUN chmod -R g=u /pulsar
### Create 2nd stage from OpenJDK image
### and add Python dependencies (for Pulsar functions)
FROM openjdk:11-jdk-slim
+# Create the pulsar group and user to make docker container run as a non root
user by default
+RUN groupadd -g 10001 pulsar
+RUN adduser -u 1 --gid 10001 --disabled-login --disabled-password --gecos
[pulsar] 12/14: Revert "Regenerate test certificates in pulsar-broker/src/test/resources/authentication/tls"
This is an automated email from the ASF dual-hosted git repository. eolivelli pushed a commit to branch 2.7.2_ds_rootless in repository https://gitbox.apache.org/repos/asf/pulsar.git commit 0bfe3e03370dc7a17dfcfc5c82e089b81b4e0623 Author: Lari Hotari AuthorDate: Fri May 14 13:05:59 2021 +0300 Revert "Regenerate test certificates in pulsar-broker/src/test/resources/authentication/tls" This reverts commit c96e31f85c0bd5c00f5692cbf61bc8ded330eddc. --- build/regenerate_certs_for_tests.sh| 7 -- .../resources/authentication/tls/broker-cert.pem | 87 .../test/resources/authentication/tls/cacert.pem | 109 + .../resources/authentication/tls/client-cert.pem | 87 4 files changed, 139 insertions(+), 151 deletions(-) diff --git a/build/regenerate_certs_for_tests.sh b/build/regenerate_certs_for_tests.sh index 372de32..7e4cf84 100755 --- a/build/regenerate_certs_for_tests.sh +++ b/build/regenerate_certs_for_tests.sh @@ -45,13 +45,6 @@ reissue_certificate $ROOT_DIR/pulsar-proxy/src/test/resources/authentication/tls $ROOT_DIR/pulsar-proxy/src/test/resources/authentication/tls/server-cert.pem generate_ca -cp ca-cert.pem $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/cacert.pem -reissue_certificate $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/broker-key.pem \ - $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem -reissue_certificate $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/client-key.pem \ - $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/client-cert.pem - -generate_ca cp ca-cert.pem $ROOT_DIR/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/cacert.pem reissue_certificate $ROOT_DIR/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/broker-key.pem \ $ROOT_DIR/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/broker-cert.pem diff --git a/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem b/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem index 66a0fb1..69ad71c 100644 --- a/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem +++ b/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem @@ -1,18 +1,18 @@ Certificate: Data: -Version: 1 (0x0) +Version: 3 (0x2) Serial Number: -03:c7:fe:e2:a7:5d:92:28:ef:5b:4f:41:1a:52:c9:12:32:75:29:a4 -Signature Algorithm: sha256WithRSAEncryption -Issuer: CN = CARoot +88:08:98:b3:13:d8:00:97 +Signature Algorithm: sha1WithRSAEncryption +Issuer: C=US, ST=CA, O=Apache, OU=Pulsar Incubator, CN=localhost Validity -Not Before: May 14 09:17:36 2021 GMT -Not After : May 12 09:17:36 2031 GMT -Subject: C = US, ST = CA, O = Apache, OU = Apache Pulsar, CN = localhost +Not Before: Feb 17 02:06:21 2018 GMT +Not After : Nov 16 00:00:00 2030 GMT +Subject: C=US, ST=CA, O=Apache, OU=Apache Pulsar, CN=localhost Subject Public Key Info: Public Key Algorithm: rsaEncryption -RSA Public-Key: (2048 bit) -Modulus: +RSA Public Key: (2048 bit) +Modulus (2048 bit): 00:af:bf:b7:2d:98:ad:9d:f6:da:a3:13:d4:62:0f: 98:be:1c:a2:89:22:ba:6f:d5:fd:1f:67:e3:91:03: 98:80:81:0e:ed:d8:f6:70:7f:2c:36:68:3d:53:ea: @@ -32,37 +32,42 @@ Certificate: a0:1a:81:9d:d2:e1:66:dd:c4:cc:fc:63:04:ac:ec: a7:35 Exponent: 65537 (0x10001) -Signature Algorithm: sha256WithRSAEncryption - aa:27:c7:ef:a4:d0:60:25:8c:bd:a1:27:58:fe:22:73:36:6b: - 98:50:d8:11:8c:0d:be:28:08:4c:ba:5c:c6:40:65:af:33:1b: - d2:a2:eb:23:02:a9:1a:f0:02:55:82:57:a2:c0:3b:1d:00:21: - f6:31:0d:8c:61:f3:72:c4:ba:87:46:db:7c:7f:62:7a:30:1b: - bf:de:d3:c5:b3:74:21:f4:30:29:d8:ba:74:d8:4a:4c:12:d4: - b9:ea:62:d1:7b:0f:e4:da:7e:76:02:5c:05:1c:a1:bf:69:7b: - 0a:67:46:a0:1e:ba:d8:3d:d6:d4:52:7c:02:87:08:7b:1d:65: - 54:03:c1:df:40:52:04:8e:58:b1:0a:6f:4c:2d:12:61:cb:34: - e2:d7:70:65:08:cb:4c:f4:ed:20:ef:e6:b4:76:16:06:f8:01: - 45:b0:10:ee:db:a1:bd:4d:57:17:f9:bd:85:ff:80:15:02:03: - 2b:ec:1d:9f:b6:2c:d7:e2:99:09:cd:09:c3:08:86:6e:ba:f8: - 2d:48:59:cf:69:f9:2c:a0:e4:81:d3:32:48:15:7a:a2:58:b3: - f7:8f:c8:7f:74:29:e0:7c:c5:ae:bf:c5:80:b4:e2:3e:d2:ec: - 07:ce:08:79:32:0b:86:6f:4f:2f:37:8d:75:d4:fc:40:38:31: - 32:88:09:36 +X509v3 extensions: +X509v3 Basic Constraints: +CA:FALSE +Netscape Comment: +OpenSSL Generated Certificate +X509v3 Subject Key Identifier: +
[pulsar] 09/14: Regenerate test certificates in pulsar-broker/src/test/resources/authentication/tls
This is an automated email from the ASF dual-hosted git repository. eolivelli pushed a commit to branch 2.7.2_ds_rootless in repository https://gitbox.apache.org/repos/asf/pulsar.git commit c96e31f85c0bd5c00f5692cbf61bc8ded330eddc Author: Lari Hotari AuthorDate: Fri May 14 12:19:36 2021 +0300 Regenerate test certificates in pulsar-broker/src/test/resources/authentication/tls --- build/regenerate_certs_for_tests.sh| 7 ++ .../resources/authentication/tls/broker-cert.pem | 87 .../test/resources/authentication/tls/cacert.pem | 109 - .../resources/authentication/tls/client-cert.pem | 87 4 files changed, 151 insertions(+), 139 deletions(-) diff --git a/build/regenerate_certs_for_tests.sh b/build/regenerate_certs_for_tests.sh index 7e4cf84..372de32 100755 --- a/build/regenerate_certs_for_tests.sh +++ b/build/regenerate_certs_for_tests.sh @@ -45,6 +45,13 @@ reissue_certificate $ROOT_DIR/pulsar-proxy/src/test/resources/authentication/tls $ROOT_DIR/pulsar-proxy/src/test/resources/authentication/tls/server-cert.pem generate_ca +cp ca-cert.pem $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/cacert.pem +reissue_certificate $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/broker-key.pem \ + $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem +reissue_certificate $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/client-key.pem \ + $ROOT_DIR/pulsar-broker/src/test/resources/authentication/tls/client-cert.pem + +generate_ca cp ca-cert.pem $ROOT_DIR/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/cacert.pem reissue_certificate $ROOT_DIR/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/broker-key.pem \ $ROOT_DIR/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/broker-cert.pem diff --git a/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem b/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem index 69ad71c..66a0fb1 100644 --- a/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem +++ b/pulsar-broker/src/test/resources/authentication/tls/broker-cert.pem @@ -1,18 +1,18 @@ Certificate: Data: -Version: 3 (0x2) +Version: 1 (0x0) Serial Number: -88:08:98:b3:13:d8:00:97 -Signature Algorithm: sha1WithRSAEncryption -Issuer: C=US, ST=CA, O=Apache, OU=Pulsar Incubator, CN=localhost +03:c7:fe:e2:a7:5d:92:28:ef:5b:4f:41:1a:52:c9:12:32:75:29:a4 +Signature Algorithm: sha256WithRSAEncryption +Issuer: CN = CARoot Validity -Not Before: Feb 17 02:06:21 2018 GMT -Not After : Nov 16 00:00:00 2030 GMT -Subject: C=US, ST=CA, O=Apache, OU=Apache Pulsar, CN=localhost +Not Before: May 14 09:17:36 2021 GMT +Not After : May 12 09:17:36 2031 GMT +Subject: C = US, ST = CA, O = Apache, OU = Apache Pulsar, CN = localhost Subject Public Key Info: Public Key Algorithm: rsaEncryption -RSA Public Key: (2048 bit) -Modulus (2048 bit): +RSA Public-Key: (2048 bit) +Modulus: 00:af:bf:b7:2d:98:ad:9d:f6:da:a3:13:d4:62:0f: 98:be:1c:a2:89:22:ba:6f:d5:fd:1f:67:e3:91:03: 98:80:81:0e:ed:d8:f6:70:7f:2c:36:68:3d:53:ea: @@ -32,42 +32,37 @@ Certificate: a0:1a:81:9d:d2:e1:66:dd:c4:cc:fc:63:04:ac:ec: a7:35 Exponent: 65537 (0x10001) -X509v3 extensions: -X509v3 Basic Constraints: -CA:FALSE -Netscape Comment: -OpenSSL Generated Certificate -X509v3 Subject Key Identifier: -D3:F3:19:AE:74:B1:AF:E7:AF:08:7B:16:72:78:29:87:79:ED:30:8C -X509v3 Authority Key Identifier: - keyid:D4:7A:CD:0F:44:1B:16:29:25:14:ED:A2:EF:13:0F:A7:46:09:78:F6 - -Signature Algorithm: sha1WithRSAEncryption -0f:04:f3:91:f2:87:19:fe:9d:f8:34:5a:24:4a:00:d1:58:bf: -1e:b2:77:67:07:bc:78:b5:4b:9a:4b:fd:a1:e5:dc:0e:09:84: -9e:59:c4:dd:cf:f7:2e:bf:da:f3:31:36:6b:81:6e:a2:88:76: -e4:2e:0b:36:44:82:36:8f:80:93:f4:9e:fc:ed:85:d0:97:da: -0f:fb:c9:b9:8b:da:ae:07:3d:4f:82:b7:0c:25:22:63:12:6b: -0a:e9:c4:12:a4:5c:ed:11:12:cc:fe:b0:2e:d4:c1:ec:79:01: -60:ea:cc:cc:e5:66:cc:57:f6:55:a9:09:4c:63:01:e9:b4:2e: -73:a5 +Signature Algorithm: sha256WithRSAEncryption + aa:27:c7:ef:a4:d0:60:25:8c:bd:a1:27:58:fe:22:73:36:6b: + 98:50:d8:11:8c:0d:be:28:08:4c:ba:5c:c6:40:65:af:33:1b: + d2:a2:eb:23:02:a9:1a:f0:02:55:82:57:a2:c0:3b:1d:00:21: + f6:31:0d:8c:61:f3:72:c4:ba:87:46:db:7c:7f:62:7a:30:1b: + bf:de:d3:c5:b3:74:21:f4:30:29:d8:ba:74:d8:4a:4c:12:d4: +
[pulsar] 07/14: Fix KeyStoreTlsTest on JDK11 (#10345)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 8c3787fadc7813c6e2875b20c2b9793315bf8186
Author: Lari Hotari
AuthorDate: Mon Apr 26 00:57:41 2021 +0300
Fix KeyStoreTlsTest on JDK11 (#10345)
---
.../apache/pulsar/client/impl/KeyStoreTlsTest.java | 8 ++--
.../util/keystoretls/KeyStoreSSLContext.java | 12 --
.../keystoretls/SSLContextValidatorEngine.java | 46 +++---
3 files changed, 26 insertions(+), 40 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/KeyStoreTlsTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/KeyStoreTlsTest.java
index 0f9993d..2746c8e 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/KeyStoreTlsTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/KeyStoreTlsTest.java
@@ -19,9 +19,7 @@
package org.apache.pulsar.client.impl;
import static org.apache.pulsar.common.util.SecurityUtility.getProvider;
-
import java.security.Provider;
-import javax.net.ssl.SSLContext;
import org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext;
import org.apache.pulsar.common.util.keystoretls.SSLContextValidatorEngine;
import org.testng.annotations.Test;
@@ -59,7 +57,7 @@ public class KeyStoreTlsTest {
true,
null,
null);
-SSLContext serverCnx = serverSSLContext.createSSLContext();
+serverSSLContext.createSSLContext();
KeyStoreSSLContext clientSSLContext = new
KeyStoreSSLContext(KeyStoreSSLContext.Mode.CLIENT,
null,
@@ -73,8 +71,8 @@ public class KeyStoreTlsTest {
false,
null,
null);
-SSLContext clientCnx = clientSSLContext.createSSLContext();
+clientSSLContext.createSSLContext();
-SSLContextValidatorEngine.validate(clientCnx, serverCnx);
+SSLContextValidatorEngine.validate(clientSSLContext::createSSLEngine,
serverSSLContext::createSSLEngine);
}
}
diff --git
a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
index c7b4cfe..e3cb6e2 100644
---
a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
+++
b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
@@ -78,7 +78,6 @@ public class KeyStoreSSLContext {
private boolean needClientAuth;
private Set ciphers;
private Set protocols;
-@Getter
private SSLContext sslContext;
private String protocol = DEFAULT_SSL_PROTOCOL;
@@ -170,12 +169,19 @@ public class KeyStoreSSLContext {
return sslContext;
}
+public SSLContext getSslContext() {
+if (sslContext == null) {
+throw new IllegalStateException("createSSLContext hasn't been
called.");
+}
+return sslContext;
+}
+
public SSLEngine createSSLEngine() {
-return configureSSLEngine(sslContext.createSSLEngine());
+return configureSSLEngine(getSslContext().createSSLEngine());
}
public SSLEngine createSSLEngine(String peerHost, int peerPort) {
-return configureSSLEngine(sslContext.createSSLEngine(peerHost,
peerPort));
+return configureSSLEngine(getSslContext().createSSLEngine(peerHost,
peerPort));
}
private SSLEngine configureSSLEngine(SSLEngine sslEngine) {
diff --git
a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/SSLContextValidatorEngine.java
b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/SSLContextValidatorEngine.java
index 555d96e..7c2f518 100644
---
a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/SSLContextValidatorEngine.java
+++
b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/SSLContextValidatorEngine.java
@@ -18,12 +18,11 @@
*/
package org.apache.pulsar.common.util.keystoretls;
+import static javax.net.ssl.SSLEngineResult.HandshakeStatus.FINISHED;
import java.nio.ByteBuffer;
-import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLEngine;
import javax.net.ssl.SSLEngineResult;
import javax.net.ssl.SSLException;
-import javax.net.ssl.SSLParameters;
import lombok.extern.slf4j.Slf4j;
/**
@@ -31,12 +30,9 @@ import lombok.extern.slf4j.Slf4j;
*/
@Slf4j
public class SSLContextValidatorEngine {
-/**
- * Mode of peer.
- */
-public enum Mode {
-CLIENT,
-SERVER
+@FunctionalInterface
+public interface SSLEngineProvider {
+SSLEngine createSSLEngine(String peerHost, int peerPort);
}
private static final ByteBuffer EMPTY_BUF = ByteBuffer.allocate(0);
@@ -44,11 +40,12 @@ public class
[pulsar] 08/14: [Tests] Fix ProxyWithAuthorizationTest on JDK11 (#10346)
This is an automated email from the ASF dual-hosted git repository. eolivelli pushed a commit to branch 2.7.2_ds_rootless in repository https://gitbox.apache.org/repos/asf/pulsar.git commit f17c5d6e8d5d051b6b65cbc0ca187f126faabcf2 Author: Lari Hotari AuthorDate: Fri Apr 23 22:28:52 2021 +0300 [Tests] Fix ProxyWithAuthorizationTest on JDK11 (#10346) * Add script for regenerating the PEM certificates * Re-create test certificates --- .../resources/authentication/tls/broker-cert.pem | 87 .../test/resources/authentication/tls/cacert.pem | 109 - .../resources/authentication/tls/client-cert.pem | 87 build/regenerate_certs_for_tests.sh| 73 ++ .../ProxyWithAuthorizationTest/broker-cacert.pem | 109 - .../tls/ProxyWithAuthorizationTest/broker-cert.pem | 86 .../ProxyWithAuthorizationTest/client-cacert.pem | 109 - .../tls/ProxyWithAuthorizationTest/client-cert.pem | 86 .../ProxyWithAuthorizationTest/proxy-cacert.pem| 109 - .../tls/ProxyWithAuthorizationTest/proxy-cert.pem | 86 .../test/resources/authentication/tls/cacert.pem | 109 - .../resources/authentication/tls/client-cert.pem | 87 .../resources/authentication/tls/server-cert.pem | 87 13 files changed, 670 insertions(+), 554 deletions(-) diff --git a/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/broker-cert.pem b/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/broker-cert.pem index 69ad71c..7f9effa 100644 --- a/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/broker-cert.pem +++ b/bouncy-castle/bcfips-include-test/src/test/resources/authentication/tls/broker-cert.pem @@ -1,18 +1,18 @@ Certificate: Data: -Version: 3 (0x2) +Version: 1 (0x0) Serial Number: -88:08:98:b3:13:d8:00:97 -Signature Algorithm: sha1WithRSAEncryption -Issuer: C=US, ST=CA, O=Apache, OU=Pulsar Incubator, CN=localhost +0c:26:15:df:8f:71:1d:6a:31:d0:da:af:64:ef:80:de:ac:9a:46:76 +Signature Algorithm: sha256WithRSAEncryption +Issuer: CN = CARoot Validity -Not Before: Feb 17 02:06:21 2018 GMT -Not After : Nov 16 00:00:00 2030 GMT -Subject: C=US, ST=CA, O=Apache, OU=Apache Pulsar, CN=localhost +Not Before: Apr 23 17:08:51 2021 GMT +Not After : Apr 21 17:08:51 2031 GMT +Subject: C = US, ST = CA, O = Apache, OU = Apache Pulsar, CN = localhost Subject Public Key Info: Public Key Algorithm: rsaEncryption -RSA Public Key: (2048 bit) -Modulus (2048 bit): +RSA Public-Key: (2048 bit) +Modulus: 00:af:bf:b7:2d:98:ad:9d:f6:da:a3:13:d4:62:0f: 98:be:1c:a2:89:22:ba:6f:d5:fd:1f:67:e3:91:03: 98:80:81:0e:ed:d8:f6:70:7f:2c:36:68:3d:53:ea: @@ -32,42 +32,37 @@ Certificate: a0:1a:81:9d:d2:e1:66:dd:c4:cc:fc:63:04:ac:ec: a7:35 Exponent: 65537 (0x10001) -X509v3 extensions: -X509v3 Basic Constraints: -CA:FALSE -Netscape Comment: -OpenSSL Generated Certificate -X509v3 Subject Key Identifier: -D3:F3:19:AE:74:B1:AF:E7:AF:08:7B:16:72:78:29:87:79:ED:30:8C -X509v3 Authority Key Identifier: - keyid:D4:7A:CD:0F:44:1B:16:29:25:14:ED:A2:EF:13:0F:A7:46:09:78:F6 - -Signature Algorithm: sha1WithRSAEncryption -0f:04:f3:91:f2:87:19:fe:9d:f8:34:5a:24:4a:00:d1:58:bf: -1e:b2:77:67:07:bc:78:b5:4b:9a:4b:fd:a1:e5:dc:0e:09:84: -9e:59:c4:dd:cf:f7:2e:bf:da:f3:31:36:6b:81:6e:a2:88:76: -e4:2e:0b:36:44:82:36:8f:80:93:f4:9e:fc:ed:85:d0:97:da: -0f:fb:c9:b9:8b:da:ae:07:3d:4f:82:b7:0c:25:22:63:12:6b: -0a:e9:c4:12:a4:5c:ed:11:12:cc:fe:b0:2e:d4:c1:ec:79:01: -60:ea:cc:cc:e5:66:cc:57:f6:55:a9:09:4c:63:01:e9:b4:2e: -73:a5 +Signature Algorithm: sha256WithRSAEncryption + 3a:38:c8:85:48:ed:84:c9:f4:bc:ef:b4:4b:a1:46:9c:97:9b: + 5f:7e:1a:ff:9b:dc:93:0e:7e:ab:de:09:21:30:1f:7f:2a:f7: + 94:d1:b3:07:3d:b1:71:4f:72:90:1f:41:3d:fe:34:14:ac:5a: + 39:02:f1:a4:8a:d1:d3:c0:48:da:6f:37:dc:b5:1d:60:29:e6: + c5:b0:ce:b4:52:8d:f6:6b:59:0b:e4:c8:f1:1a:40:3a:4f:bd: + e2:dd:32:2f:21:3c:33:d7:61:5f:86:cd:94:31:31:f1:ff:c6: + 08:9e:67:bc:8f:9d:bf:38:a8:8c:ff:3f:1f:fb:24:ab:bb:7c: + fb:1b:c3:1b:62:b4:dd:21:d3:7b:19:92:16:b7:7d:f6:95:ee: + 14:a0:83:de:c5:05:d8:af:44:1d:f7:eb:32:e2:03:ac:c9:12: + df:11:b6:af:f8:b9:24:ae:55:3e:25:ae:2a:b2:d3:b6:6a:e9: +
[pulsar] 05/14: Remove solr
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit cfadf050bbb5adde2d01518affe98c98f4a4a9c4
Author: Enrico Olivelli
AuthorDate: Thu May 13 15:50:16 2021 +0200
Remove solr
---
distribution/io/src/assemble/io.xml | 1 -
1 file changed, 1 deletion(-)
diff --git a/distribution/io/src/assemble/io.xml
b/distribution/io/src/assemble/io.xml
index 1ae97bf..61626fc 100644
--- a/distribution/io/src/assemble/io.xml
+++ b/distribution/io/src/assemble/io.xml
@@ -74,7 +74,6 @@
${basedir}/../../pulsar-io/influxdb/target/pulsar-io-influxdb-${project.version}.nar
${basedir}/../../pulsar-io/redis/target/pulsar-io-redis-${project.version}.nar
${basedir}/../../pulsar-io/flume/target/pulsar-io-flume-${project.version}.nar
-
${basedir}/../../pulsar-io/solr/target/pulsar-io-solr-${project.version}.nar
${basedir}/../../pulsar-io/dynamodb/target/pulsar-io-dynamodb-${project.version}.nar
[pulsar] 04/14: Remove Solr
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 53120ffa6799fc4b9883572b8f8d286856db0244
Author: Enrico Olivelli
AuthorDate: Thu May 13 14:26:10 2021 +0200
Remove Solr
---
pulsar-io/pom.xml | 1 -
pulsar-io/solr/pom.xml | 86 ---
.../apache/pulsar/io/solr/SolrAbstractSink.java| 131
.../pulsar/io/solr/SolrGenericRecordSink.java | 53 ---
.../org/apache/pulsar/io/solr/SolrSinkConfig.java | 98
.../resources/META-INF/services/pulsar-io.yaml | 23 ---
.../pulsar/io/solr/SolrGenericRecordSinkTest.java | 114 --
.../org/apache/pulsar/io/solr/SolrServerUtil.java | 91 ---
.../apache/pulsar/io/solr/SolrSinkConfigTest.java | 168 -
pulsar-io/solr/src/test/resources/sinkConfig.yaml | 27
pulsar-io/solr/src/test/resources/solr.xml | 38 -
11 files changed, 830 deletions(-)
diff --git a/pulsar-io/pom.xml b/pulsar-io/pom.xml
index 40b476a..a3911c0 100644
--- a/pulsar-io/pom.xml
+++ b/pulsar-io/pom.xml
@@ -64,7 +64,6 @@
mongo
flume
redis
-solr
influxdb
dynamodb
nsq
diff --git a/pulsar-io/solr/pom.xml b/pulsar-io/solr/pom.xml
deleted file mode 100644
index a8d6103..000
--- a/pulsar-io/solr/pom.xml
+++ /dev/null
@@ -1,86 +0,0 @@
-
-http://maven.apache.org/POM/4.0.0;
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
-4.0.0
-
-pulsar-io
-org.apache.pulsar
-2.7.2.1.0.0
-
-
-
-8.6.3
-
-
-pulsar-io-solr
-Pulsar IO :: Solr
-
-
-
-${project.parent.groupId}
-pulsar-io-core
-${project.parent.version}
-
-
-${project.groupId}
-pulsar-functions-instance
-${project.version}
-
-
-${project.groupId}
-pulsar-client-original
-${project.version}
-
-
-org.apache.solr
-solr-solrj
-${solr.version}
-
-
-org.apache.solr
-solr-core
-${solr.version}
-test
-
-
-org.apache.commons
-commons-lang3
-3.4
-
-
-commons-collections
-commons-collections
-3.2.2
-
-
-
-
-
-
-org.apache.nifi
-nifi-nar-maven-plugin
-
-
-
-
-
\ No newline at end of file
diff --git
a/pulsar-io/solr/src/main/java/org/apache/pulsar/io/solr/SolrAbstractSink.java
b/pulsar-io/solr/src/main/java/org/apache/pulsar/io/solr/SolrAbstractSink.java
deleted file mode 100644
index d7bcb12..000
---
a/pulsar-io/solr/src/main/java/org/apache/pulsar/io/solr/SolrAbstractSink.java
+++ /dev/null
@@ -1,131 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pulsar.io.solr;
-
-import com.google.common.base.Strings;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.pulsar.functions.api.Record;
-import org.apache.pulsar.io.core.Sink;
-import org.apache.pulsar.io.core.SinkContext;
-import org.apache.solr.client.solrj.SolrClient;
-import org.apache.solr.client.solrj.SolrServerException;
-import org.apache.solr.client.solrj.impl.CloudSolrClient;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
-import org.apache.solr.client.solrj.request.UpdateRequest;
-import org.apache.solr.client.solrj.response.UpdateResponse;
-import org.apache.solr.common.SolrInputDocument;
-
-import java.io.IOException;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
-import java.util.Optional;
-
-/**
- * A simple abstract class for Solr sink
- */
-@Slf4j
-public abstract class SolrAbstractSink implements Sink {
-
-private SolrSinkConfig solrSinkConfig;
-
[pulsar] 06/14: Use Message.getReaderSchema() in Pulsar IO Sinks when possible (#10557)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 5dee3ebe51ee44c859c96f580ddad4769b9f0895
Author: Enrico Olivelli
AuthorDate: Fri May 14 07:29:53 2021 +0200
Use Message.getReaderSchema() in Pulsar IO Sinks when possible (#10557)
(cherry picked from commit 90117b2be8df9893e0f9a7b6829d48bdddc7c55f)
---
.../apache/pulsar/client/api/SimpleSchemaTest.java | 5 ++
.../client/impl/schema/AbstractStructSchema.java | 8 ++
.../pulsar/client/impl/schema/KeyValueSchema.java | 10 ++-
.../pulsar/functions/instance/SinkRecord.java | 12 ++-
.../integration/io/TestGenericObjectSink.java | 15 +++-
.../io/PulsarGenericObjectSinkTest.java| 86 +-
6 files changed, 131 insertions(+), 5 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
index d594688..af49f96 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
@@ -616,6 +616,11 @@ public class SimpleSchemaTest extends ProducerConsumerBase
{
assertEquals(data.getKey().getField("i"), i * 100);
assertEquals(data.getValue().getField("i"), i * 1000);
c0.acknowledge(wrapper);
+Schema schema = wrapper.getReaderSchema().get();
+KeyValueSchema keyValueSchema = (KeyValueSchema) schema;
+assertEquals(SchemaType.AVRO,
keyValueSchema.getKeySchema().getSchemaInfo().getType());
+assertEquals(SchemaType.AVRO,
keyValueSchema.getValueSchema().getSchemaInfo().getType());
+assertNotNull(schema.getSchemaInfo());
}
// verify c1
for (int i = 0; i < numMessages; i++) {
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractStructSchema.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractStructSchema.java
index ce7..ce68434 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractStructSchema.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractStructSchema.java
@@ -158,6 +158,14 @@ public abstract class AbstractStructSchema extends
AbstractSchema {
return Optional.empty();
}
}
+
+@Override
+public String toString() {
+return "VersionedSchema(type=" + schemaInfo.getType() +
+",schemaVersion="+BytesSchemaVersion.of(schemaVersion) +
+",name="+schemaInfo.getName()
++ ")";
+}
}
private AbstractStructSchema getAbstractStructSchemaAtVersion(byte[]
schemaVersion, SchemaInfo schemaInfo) {
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/KeyValueSchema.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/KeyValueSchema.java
index f572bbf..c33de77 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/KeyValueSchema.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/KeyValueSchema.java
@@ -122,6 +122,8 @@ public class KeyValueSchema extends
AbstractSchema> {
// defer configuring the key/value schema info until
`configureSchemaInfo` is called.
if (!requireFetchingSchemaInfo()) {
configureKeyValueSchemaInfo();
+} else {
+buildKeyValueSchemaInfo();
}
}
@@ -224,10 +226,14 @@ public class KeyValueSchema extends
AbstractSchema> {
return KeyValueSchema.of(keySchema.clone(), valueSchema.clone(),
keyValueEncodingType);
}
-private void configureKeyValueSchemaInfo() {
+private void buildKeyValueSchemaInfo() {
this.schemaInfo = KeyValueSchemaInfo.encodeKeyValueSchemaInfo(
-keySchema, valueSchema, keyValueEncodingType
+keySchema, valueSchema, keyValueEncodingType
);
+}
+
+private void configureKeyValueSchemaInfo() {
+buildKeyValueSchemaInfo();
this.keySchema.setSchemaInfoProvider(new SchemaInfoProvider() {
@Override
public CompletableFuture getSchemaByVersion(byte[]
schemaVersion) {
diff --git
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/SinkRecord.java
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/SinkRecord.java
index 38c7036..a7ff0eb 100644
---
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/SinkRecord.java
+++
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/SinkRecord.java
@@ -98,8 +98,18 @@
[pulsar] 03/14: Allow to build Pulsar with JDK11 and -Dmaven.compiler.release=8 (#9580)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 7e1f9f06f2a4dd031169a8890151eda8042bfc0b
Author: Enrico Olivelli
AuthorDate: Fri Feb 12 21:55:32 2021 +0100
Allow to build Pulsar with JDK11 and -Dmaven.compiler.release=8 (#9580)
* Allow to build Pulsar with JDK11 and -Dmaven.compiler.release=8
* fix style
Co-authored-by: Enrico Olivelli
(cherry picked from commit 731c18f5b98513ca9018e45ed1fdeff298a5312b)
---
.../pulsar/client/impl/MultiMessageIdImpl.java | 3 +-
.../common/stats/JvmDefaultGCMetricsLogger.java| 45 ++
.../stats/JvmDefaultGCMetricsLoggerTest.java | 39 +++
3 files changed, 78 insertions(+), 9 deletions(-)
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiMessageIdImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiMessageIdImpl.java
index e6e7557..dfe995e 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiMessageIdImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiMessageIdImpl.java
@@ -23,7 +23,6 @@ import java.util.Map.Entry;
import java.util.Objects;
import lombok.Getter;
import org.apache.pulsar.client.api.MessageId;
-import sun.reflect.generics.reflectiveObjects.NotImplementedException;
/**
* A MessageId implementation that contains a map of .
@@ -42,7 +41,7 @@ public class MultiMessageIdImpl implements MessageId {
// https://github.com/apache/pulsar/issues/4940
@Override
public byte[] toByteArray() {
-throw new NotImplementedException();
+throw new UnsupportedOperationException();
}
@Override
diff --git
a/pulsar-common/src/main/java/org/apache/pulsar/common/stats/JvmDefaultGCMetricsLogger.java
b/pulsar-common/src/main/java/org/apache/pulsar/common/stats/JvmDefaultGCMetricsLogger.java
index 307daa1..4726a2c 100644
---
a/pulsar-common/src/main/java/org/apache/pulsar/common/stats/JvmDefaultGCMetricsLogger.java
+++
b/pulsar-common/src/main/java/org/apache/pulsar/common/stats/JvmDefaultGCMetricsLogger.java
@@ -19,14 +19,18 @@
package org.apache.pulsar.common.stats;
import com.google.common.collect.Maps;
+
import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;
+import java.lang.reflect.Method;
import java.util.List;
import java.util.Map;
+
+import lombok.SneakyThrows;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-@SuppressWarnings({"restriction", "checkstyle:JavadocType"})
+@SuppressWarnings({"checkstyle:JavadocType"})
public class JvmDefaultGCMetricsLogger implements JvmGCMetricsLogger {
private static final Logger log =
LoggerFactory.getLogger(JvmDefaultGCMetricsLogger.class);
@@ -36,19 +40,46 @@ public class JvmDefaultGCMetricsLogger implements
JvmGCMetricsLogger {
private volatile long accumulatedFullGcTime = 0;
private volatile long currentFullGcTime = 0;
-@SuppressWarnings("restriction")
-private static sun.management.HotspotRuntimeMBean runtime;
+private static Object /*sun.management.HotspotRuntimeMBean*/ runtime;
+private static Method getTotalSafepointTimeHandle;
+private static Method getSafepointCountHandle;
private Map gcMetricsMap = Maps.newHashMap();
static {
try {
-runtime =
sun.management.ManagementFactoryHelper.getHotspotRuntimeMBean();
-} catch (Exception e) {
+runtime = Class.forName("sun.management.ManagementFactoryHelper")
+.getMethod("getHotspotRuntimeMBean")
+.invoke(null);
+getTotalSafepointTimeHandle =
runtime.getClass().getMethod("getTotalSafepointTime");
+getTotalSafepointTimeHandle.setAccessible(true);
+getSafepointCountHandle =
runtime.getClass().getMethod("getSafepointCount");
+getSafepointCountHandle.setAccessible(true);
+
+// try to use the methods
+getTotalSafepointTimeHandle.invoke(runtime);
+getSafepointCountHandle.invoke(runtime);
+} catch (Throwable e) {
log.warn("Failed to get Runtime bean", e);
}
}
+@SneakyThrows
+static long getTotalSafepointTime() {
+if (getTotalSafepointTimeHandle == null) {
+return -1;
+}
+return (long) getTotalSafepointTimeHandle.invoke(runtime);
+}
+
+@SneakyThrows
+static long getSafepointCount() {
+if (getTotalSafepointTimeHandle == null) {
+return -1;
+}
+return (long) getSafepointCountHandle.invoke(runtime);
+}
+
/**
* Metrics for the Garbage Collector.
*/
@@ -92,8 +123,8 @@ public class JvmDefaultGCMetricsLogger implements
JvmGCMetricsLogger {
* that the application has been stopped
[pulsar] 02/14: [Tests] Recreate keystores used in TLS tests with RSA key algorithm & SHA256 to support JDK 11 & TLS 1.3 (#10336)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 47ce9dee6311d0174bd1cf2c05456daa9d9c578f
Author: Lari Hotari
AuthorDate: Mon Apr 26 20:18:27 2021 +0300
[Tests] Recreate keystores used in TLS tests with RSA key algorithm &
SHA256 to support JDK 11 & TLS 1.3 (#10336)
* Add script for creating certs for tests
* RSA keys must be used, update documentation for creating keys with keytool
* Update keystores used in tests
(cherry picked from commit 0e4ff8a4a414f02b44a027ead01f49d8b3ab2ade)
---
build/generate_keystores_for_tests.sh | 61 +
.../authentication/keystoretls/broker.keystore.jks | Bin 2767 -> 3723 bytes
.../keystoretls/broker.truststore.jks | Bin 731 -> 838 bytes
.../authentication/keystoretls/client.keystore.jks | Bin 2767 -> 3726 bytes
.../keystoretls/client.truststore.jks | Bin 731 -> 838 bytes
.../authentication/keystoretls/broker.keystore.jks | Bin 2767 -> 3723 bytes
.../keystoretls/broker.truststore.jks | Bin 731 -> 838 bytes
.../authentication/keystoretls/client.keystore.jks | Bin 2767 -> 3726 bytes
.../keystoretls/client.truststore.jks | Bin 731 -> 838 bytes
site2/docs/security-tls-keystore.md| 2 +-
.../version-2.6.0/security-tls-keystore.md | 2 +-
.../version-2.6.1/security-tls-keystore.md | 2 +-
.../version-2.6.2/security-tls-keystore.md | 2 +-
.../security-tls-keystore.md | 8 +--
.../security-tls-keystore.md | 4 +-
.../security-tls-keystore.md | 8 +--
.../security-tls-keystore.md | 8 +--
17 files changed, 79 insertions(+), 18 deletions(-)
diff --git a/build/generate_keystores_for_tests.sh
b/build/generate_keystores_for_tests.sh
new file mode 100755
index 000..551ce8e
--- /dev/null
+++ b/build/generate_keystores_for_tests.sh
@@ -0,0 +1,61 @@
+#!/bin/bash -xe
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+cd /tmp
+mkdir keygendir$$
+cd keygendir$$
+
+# create CA key and cert
+openssl req -x509 -newkey rsa:2048 -passout pass:11 -keyout ca-key -out
ca-cert -days 3650 -sha256 -subj "/CN=CARoot"
+
+COMMON_PARAMS="-storetype JKS -storepass 11 -keypass 11 -noprompt"
+
+# create client and broker truststores and keystores
+keytool -import -keystore client.truststore.jks $COMMON_PARAMS -alias CARoot
-file ca-cert
+keytool -import -keystore broker.truststore.jks $COMMON_PARAMS -alias CARoot
-file ca-cert
+keytool -import -keystore client.keystore.jks $COMMON_PARAMS -alias CARoot
-file ca-cert
+keytool -import -keystore broker.keystore.jks $COMMON_PARAMS -alias CARoot
-file ca-cert
+
+# create broker key
+keytool -genkeypair -keystore broker.keystore.jks $COMMON_PARAMS -keyalg RSA
-alias localhost -validity 3650 \
+ -dname 'CN=localhost,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown'
+keytool -certreq -keystore broker.keystore.jks $COMMON_PARAMS -alias localhost
-file cert-file
+# sign broker key
+openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed
-days 3650 -CAcreateserial -passin pass:11
+# import broker key
+keytool -import -keystore broker.keystore.jks $COMMON_PARAMS -alias localhost
-file cert-signed
+
+# create client key
+keytool -genkeypair -keystore client.keystore.jks $COMMON_PARAMS -keyalg RSA
-alias clientuser -validity 3650 \
+ -dname 'CN=clientuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown'
+keytool -certreq -keystore client.keystore.jks $COMMON_PARAMS -alias
clientuser -file cert-file-client
+# sign client key
+openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file-client -out
cert-signed-client -days 3650 -CAcreateserial -passin pass:11
+# import client key
+keytool -import -keystore client.keystore.jks $COMMON_PARAMS -alias clientuser
-file cert-signed-client
+
+# update keystores used in tests
+cp client.truststore.jks broker.truststore.jks
[pulsar] branch 2.7.2_ds_rootless created (now 376867a)
This is an automated email from the ASF dual-hosted git repository. eolivelli pushed a change to branch 2.7.2_ds_rootless in repository https://gitbox.apache.org/repos/asf/pulsar.git. at 376867a [Issue 8751] Update Dockerfile for Pulsar and Dashboard to Create and Use pulsar User (nonroot user) (#8796) This branch includes the following new commits: new 62624e7 Fix build new 47ce9de [Tests] Recreate keystores used in TLS tests with RSA key algorithm & SHA256 to support JDK 11 & TLS 1.3 (#10336) new 7e1f9f0 Allow to build Pulsar with JDK11 and -Dmaven.compiler.release=8 (#9580) new 53120ff Remove Solr new cfadf05 Remove solr new 5dee3eb Use Message.getReaderSchema() in Pulsar IO Sinks when possible (#10557) new 8c3787f Fix KeyStoreTlsTest on JDK11 (#10345) new f17c5d6 [Tests] Fix ProxyWithAuthorizationTest on JDK11 (#10346) new c96e31f Regenerate test certificates in pulsar-broker/src/test/resources/authentication/tls new 1a94bbc [Build] Specify release in maven-compiler-plugin configuration on JDK11 (#10343) new d5aeb66 Add AvroSchema UUID support new 0bfe3e0 Revert "Regenerate test certificates in pulsar-broker/src/test/resources/authentication/tls" new feb511d Fix expired tls certs for cpp tests (#9607) new 376867a [Issue 8751] Update Dockerfile for Pulsar and Dashboard to Create and Use pulsar User (nonroot user) (#8796) The 14 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[pulsar] 01/14: Fix build
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_rootless
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 62624e7ac74ea6dff84989cbab56ce85bc090f78
Author: Enrico Olivelli
AuthorDate: Thu May 13 10:01:04 2021 +0200
Fix build
---
.../persistent/PersistentStreamingDispatcherSingleActiveConsumer.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherSingleActiveConsumer.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherSingleActiveConsumer.java
index b4e4ed3..94ae6e7 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherSingleActiveConsumer.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherSingleActiveConsumer.java
@@ -36,7 +36,7 @@ import org.apache.pulsar.broker.service.Subscription;
import
org.apache.pulsar.broker.service.streamingdispatch.PendingReadEntryRequest;
import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
import org.apache.pulsar.broker.service.streamingdispatch.StreamingEntryReader;
-import org.apache.pulsar.common.api.proto.CommandSubscribe.SubType;
+import org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.SubType;
/**
* A {@link PersistentDispatcherSingleActiveConsumer} implemented {@link
StreamingDispatcher}.
[GitHub] [pulsar] lhotari opened a new issue #10589: Tests that don't belong to any TestNG group aren't run at all in pulsar-broker module
lhotari opened a new issue #10589: URL: https://github.com/apache/pulsar/issues/10589 **Describe the bug** There are some tests that don't belong to any TestNG group. These tests aren't currently run in CI at all. **To Reproduce** org.apache.pulsar.client.impl.MessagePublishThrottlingTest is one of such tests. There are 77 tests in total that are currently skipped since they don't belong to any group. **Expected behavior** All tests (unless marked as ignored/disabled) should get run in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Fix NPE when ACK grouping tracker checks duplicated message id (#10586)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 993ea62 Fix NPE when ACK grouping tracker checks duplicated message
id (#10586)
993ea62 is described below
commit 993ea62108ed2c21bd14e0f1f7e792e0154435b9
Author: Yunze Xu
AuthorDate: Fri May 14 19:44:27 2021 +0800
Fix NPE when ACK grouping tracker checks duplicated message id (#10586)
---
.../src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java | 3 +++
.../client/impl/PersistentAcknowledgmentsGroupingTracker.java | 6 +-
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
index 855ee64..2d57185 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
@@ -200,6 +200,9 @@ public class MessageIdImpl implements MessageId {
@Override
public int compareTo(MessageId o) {
+if (o == null) {
+throw new UnsupportedOperationException("MessageId is null");
+}
if (o instanceof MessageIdImpl) {
MessageIdImpl other = (MessageIdImpl) o;
int batchIndex = (o instanceof BatchMessageIdImpl) ?
((BatchMessageIdImpl) o).getBatchIndex() : NO_BATCH;
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
index 9eee2bf..09d6bd5 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
@@ -36,6 +36,7 @@ import
java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import io.netty.util.Recycler;
+import lombok.NonNull;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.tuple.Triple;
import org.apache.pulsar.client.api.MessageId;
@@ -113,7 +114,10 @@ public class PersistentAcknowledgmentsGroupingTracker
implements Acknowledgments
* resent after a disconnection and for which the user has already sent an
acknowledgement.
*/
@Override
-public boolean isDuplicate(MessageId messageId) {
+public boolean isDuplicate(@NonNull MessageId messageId) {
+if (lastCumulativeAck.messageId == null) {
+return false;
+}
if (messageId.compareTo(lastCumulativeAck.messageId) <= 0) {
// Already included in a cumulative ack
return true;
[GitHub] [pulsar] eolivelli merged pull request #10586: Fix NPE when ACK grouping tracker checks duplicated message id
eolivelli merged pull request #10586: URL: https://github.com/apache/pulsar/pull/10586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] KannarFr edited a comment on pull request #8048: [Issue 7903][proxy]Enable AutoTopicCreationType partitioned through proxy
KannarFr edited a comment on pull request #8048: URL: https://github.com/apache/pulsar/pull/8048#issuecomment-841179075 @rudy2steiner can you clean rebase from `apache:master`? The tests are failing in `PulsarFunctionsProcessTest` in `rudy2steiner:master` and this file no more exists in `apache:master`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] KannarFr edited a comment on pull request #8048: [Issue 7903][proxy]Enable AutoTopicCreationType partitioned through proxy
KannarFr edited a comment on pull request #8048: URL: https://github.com/apache/pulsar/pull/8048#issuecomment-841179075 @rudy2steiner can you clean rebase from `pulsar:master`? The tests are failing in `PulsarFunctionsProcessTest` and this file no more exists in `pulsar:master`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] KannarFr commented on pull request #8048: [Issue 7903][proxy]Enable AutoTopicCreationType partitioned through proxy
KannarFr commented on pull request #8048: URL: https://github.com/apache/pulsar/pull/8048#issuecomment-841179075 @rudy2steiner can you clean rebase from `pulsar:master`? The tests are failing in `PulsarFunctionsProcessTest` and this file no more exists in `apache:master`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #10586: Fix NPE when ACK grouping tracker checks duplicated message id
BewareMyPower commented on pull request #10586: URL: https://github.com/apache/pulsar/pull/10586#issuecomment-841178616 Now all tests passed, PTAL again @eolivelli -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] daniel-bartz commented on issue #10584: broker.conf documentation : wrong tokenSecretKey base64 inline description
daniel-bartz commented on issue #10584: URL: https://github.com/apache/pulsar/issues/10584#issuecomment-841174356 @Anonymitaet Yes, I tried it and it did not work for me with the semicolon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10588: When to support the springboot ecosystem
lhotari commented on issue #10588: URL: https://github.com/apache/pulsar/issues/10588#issuecomment-841150203 btw. there's a feature request #10437 which is related to supporting Spring Reactive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tuteng commented on issue #10588: When to support the springboot ecosystem
tuteng commented on issue #10588: URL: https://github.com/apache/pulsar/issues/10588#issuecomment-841149214 One community example https://github.com/majusko/pulsar-java-spring-boot-starter -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #10587: Support alias for the Schema and Message
eolivelli commented on issue #10587: URL: https://github.com/apache/pulsar/issues/10587#issuecomment-841147221 I see your problem. but we have to find some way in order to abstract from Jackson Mapper because relying on Jackson Mapper will add an hard dependency for Pulsar. in the case of Avro it is fine to support their annotations because we are supporting Apache Avro. but in this case Pulsar supports JSON in general and not Jackson Mapper, that is a specific library. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] MichealJou opened a new issue #10588: When to support the springboot ecosystem
MichealJou opened a new issue #10588: URL: https://github.com/apache/pulsar/issues/10588 When to support the springboot ecosystem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10428: [Issue #10427] Add AvroSchema UUID support fix
eolivelli commented on a change in pull request #10428:
URL: https://github.com/apache/pulsar/pull/10428#discussion_r632409675
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/util/SchemaUtil.java
##
@@ -88,8 +90,10 @@ public static Schema extractAvroSchema(SchemaDefinition
schemaDefinition, Class
try {
return
parseAvroSchema(pojo.getDeclaredField("SCHEMA$").get(null).toString());
} catch (NoSuchFieldException | IllegalAccessException |
IllegalArgumentException ignored) {
-return schemaDefinition.getAlwaysAllowNull() ?
ReflectData.AllowNull.get().getSchema(pojo)
-: ReflectData.get().getSchema(pojo);
+ReflectData reflectData = schemaDefinition.getAlwaysAllowNull() ?
ReflectData.AllowNull.get()
+: ReflectData.get();
+AvroSchema.addLogicalTypeConversions(reflectData,
schemaDefinition.isJsr310ConversionEnabled());
Review comment:
This code is executed when you build the schema from a POJO that is not
generated by Avro.
I guess that without this change the test is not passing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #10540: Apache Pulsar 2.7.2 blog page
eolivelli commented on pull request #10540: URL: https://github.com/apache/pulsar/pull/10540#issuecomment-841130367 thank you @Anonymitaet let's merge this patch **only after** the official announcement of the release -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] cydrain commented on issue #516: pulsar RACE issue
cydrain commented on issue #516: URL: https://github.com/apache/pulsar-client-go/issues/516#issuecomment-841115685 Any update ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] abhilashmandaliya opened a new issue #10587: Support alias for the Schema and Message
abhilashmandaliya opened a new issue #10587: URL: https://github.com/apache/pulsar/issues/10587 Schema and message should support different names/alias while performing SerDe. Like `ObjectMapper` has `@JsonAlias` and `@JsonProperty`. Probably we can use the same feature as Pulsar Schemas use `ObjectMapper` internally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10540: Apache Pulsar 2.7.2 blog page
eolivelli commented on a change in pull request #10540: URL: https://github.com/apache/pulsar/pull/10540#discussion_r632373985 ## File path: site2/website/blog/2021-05-13-Apache-Pulsar-2-7-2.md ## @@ -0,0 +1,68 @@ +--- +author: Enrico Olivelli +authorURL: https://twitter.com/eolivelli +title: Apache Pulsar 2.7.2 +--- +We are very glad to see the Apache Pulsar community has successfully released 2.7.2 version. +This is a minor release that introduces stability fixes and a few new features without breaking changes. + + + +### News and noteworthy + +Here is a selection of the most interesting and major features added to Pulsar 2.7.2. + +- Improvements in stability in the Kinesis Connector [#10420](https://github.com/apache/pulsar/pull/10420) +- Improvement in passing ENV variables to the Bookie (PULSAR_EXTRA_OPTS) [#10397](https://github.com/apache/pulsar/pull/10397) +- Allow the C++ client to be built in Windows and add CI for verification [#10387](https://github.com/apache/pulsar/pull/10387) +- Allow to activate every BookKeeper Client features in the Broker [#9232](https://github.com/apache/pulsar/pull/9232) +- Improvements Pulsar Proxy +- Upgrade core networking libraries: Jetty and Netty + +[Here](https://github.com/apache/pulsar/pulls?page=1=is%3Apr+label%3Arelease%2F2.7.2]) you can find the list of all the improvements and bugfixes. + +### Contributors for 2.7.2 release + +We would like to thank all the contributors for this release. +As every healthy Open Source Project, Apache Pulsar is great because it is supported by a vibrant community. + +Code contributors (names taken from GitHub API): +- Ali Ahmed +- Andrey Yegorov +- Binbin Guo +- David Kjerrumgaard +- Deon van der Vyver +- Devin Bost +- Enrico Olivelli +- Guangning E +- Kevin Wilson +- Lari Hotari +- Marvin Cai +- Masahiro Sakamoto +- Matteo Merli +- Michael Marshall +- Rajan Dhabalia +- Shen Liu +- Ting Yuan +- Vincent Royer +- Yong Zhang +- Yunze Xu +- Zhanpeng Wu +- Zike Yang +- baomingyu +- CongBo +- dockerzhang +- feynmanlin +- hangc0276 +- li jinquan +- limingnihao +- linlinnn +- mlyahmed +- PengHui Li +- ran + Review comment: @Anonymitaet this is really true indeed! How can I get a list ? Probably they have not been listed because docs changes happen mostly in the "master" branch. Can you please suggest me the names ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10540: Apache Pulsar 2.7.2 blog page
eolivelli commented on a change in pull request #10540: URL: https://github.com/apache/pulsar/pull/10540#discussion_r632372904 ## File path: site2/website/blog/2021-05-13-Apache-Pulsar-2-7-2.md ## @@ -0,0 +1,68 @@ +--- +author: Enrico Olivelli +authorURL: https://twitter.com/eolivelli +title: Apache Pulsar 2.7.2 +--- +We are very glad to see the Apache Pulsar community has successfully released 2.7.2 version. +This is a minor release that introduces stability fixes and a few new features without breaking changes. + + + +### News and noteworthy + +Here is a selection of the most interesting and major features added to Pulsar 2.7.2. + +- Improvements in stability in the Kinesis Connector [#10420](https://github.com/apache/pulsar/pull/10420) +- Improvement in passing ENV variables to the Bookie (PULSAR_EXTRA_OPTS) [#10397](https://github.com/apache/pulsar/pull/10397) +- Allow the C++ client to be built in Windows and add CI for verification [#10387](https://github.com/apache/pulsar/pull/10387) +- Allow to activate every BookKeeper Client features in the Broker [#9232](https://github.com/apache/pulsar/pull/9232) +- Improvements Pulsar Proxy +- Upgrade core networking libraries: Jetty and Netty + +[Here](https://github.com/apache/pulsar/pulls?page=1=is%3Apr+label%3Arelease%2F2.7.2]) you can find the list of all the improvements and bugfixes. + +### Contributors for 2.7.2 release + +We would like to thank all the contributors for this release. +As every healthy Open Source Project, Apache Pulsar is great because it is supported by a vibrant community. + +Code contributors (names taken from GitHub API): +- Ali Ahmed +- Andrey Yegorov +- Binbin Guo +- David Kjerrumgaard +- Deon van der Vyver +- Devin Bost +- Enrico Olivelli +- Guangning E +- Kevin Wilson +- Lari Hotari +- Marvin Cai +- Masahiro Sakamoto +- Matteo Merli +- Michael Marshall +- Rajan Dhabalia +- Shen Liu +- Ting Yuan +- Vincent Royer +- Yong Zhang +- Yunze Xu +- Zhanpeng Wu +- Zike Yang +- baomingyu +- CongBo +- dockerzhang +- feynmanlin +- hangc0276 +- li jinquan +- limingnihao +- linlinnn +- mlyahmed +- PengHui Li +- ran Review comment: good -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a change in pull request #10586: Fix NPE when ACK grouping tracker checks duplicated message id
BewareMyPower commented on a change in pull request #10586:
URL: https://github.com/apache/pulsar/pull/10586#discussion_r632372675
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
##
@@ -210,7 +210,8 @@ public int compareTo(MessageId o) {
} else if (o instanceof TopicMessageIdImpl) {
return compareTo(((TopicMessageIdImpl) o).getInnerMessageId());
} else {
-throw new UnsupportedOperationException("Unknown MessageId type: "
+ o.getClass().getName());
+final String typeName = (o != null) ? o.getClass().getName() :
"null";
Review comment:
I agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10540: Apache Pulsar 2.7.2 blog page
eolivelli commented on a change in pull request #10540: URL: https://github.com/apache/pulsar/pull/10540#discussion_r632371762 ## File path: site2/website/blog/2021-05-13-Apache-Pulsar-2-7-2.md ## @@ -0,0 +1,68 @@ +--- +author: Enrico Olivelli +authorURL: https://twitter.com/eolivelli +title: Apache Pulsar 2.7.2 +--- +We are very glad to see the Apache Pulsar community has successfully released 2.7.2 version. +This is a minor release that introduces stability fixes and a few new features without breaking changes. + + + +### News and noteworthy + +Here is a selection of the most interesting and major features added to Pulsar 2.7.2. + +- Improvements in stability in the Kinesis Connector [#10420](https://github.com/apache/pulsar/pull/10420) +- Improvement in passing ENV variables to the Bookie (PULSAR_EXTRA_OPTS) [#10397](https://github.com/apache/pulsar/pull/10397) +- Allow the C++ client to be built in Windows and add CI for verification [#10387](https://github.com/apache/pulsar/pull/10387) +- Allow to activate every BookKeeper Client features in the Broker [#9232](https://github.com/apache/pulsar/pull/9232) +- Improvements Pulsar Proxy +- Upgrade core networking libraries: Jetty and Netty + +[Here](https://github.com/apache/pulsar/pulls?page=1=is%3Apr+label%3Arelease%2F2.7.2]) you can find the list of all the improvements and bugfixes. + +### Contributors for 2.7.2 release + +We would like to thank all the contributors for this release. +As every healthy Open Source Project, Apache Pulsar is great because it is supported by a vibrant community. + +Code contributors (names taken from GitHub API): +- Ali Ahmed +- Andrey Yegorov +- Binbin Guo +- David Kjerrumgaard +- Deon van der Vyver +- Devin Bost +- Enrico Olivelli +- Guangning E +- Kevin Wilson +- Lari Hotari +- Marvin Cai +- Masahiro Sakamoto +- Matteo Merli +- Michael Marshall +- Rajan Dhabalia +- Shen Liu +- Ting Yuan +- Vincent Royer +- Yong Zhang +- Yunze Xu +- Zhanpeng Wu +- Zike Yang +- baomingyu +- CongBo +- dockerzhang +- feynmanlin +- hangc0276 +- li jinquan +- limingnihao +- linlinnn +- mlyahmed +- PengHui Li +- ran + +Also, we want to thank everyone who spent his time to report issues and to share his story about using Pulsar. Review comment: interesting. good to know. I will keep in mind for the next time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #10540: Apache Pulsar 2.7.2 blog page
Anonymitaet commented on a change in pull request #10540: URL: https://github.com/apache/pulsar/pull/10540#discussion_r632361580 ## File path: site2/website/blog/2021-05-13-Apache-Pulsar-2-7-2.md ## @@ -0,0 +1,68 @@ +--- +author: Enrico Olivelli +authorURL: https://twitter.com/eolivelli +title: Apache Pulsar 2.7.2 +--- +We are very glad to see the Apache Pulsar community has successfully released 2.7.2 version. +This is a minor release that introduces stability fixes and a few new features without breaking changes. + + + +### News and noteworthy + +Here is a selection of the most interesting and major features added to Pulsar 2.7.2. + +- Improvements in stability in the Kinesis Connector [#10420](https://github.com/apache/pulsar/pull/10420) +- Improvement in passing ENV variables to the Bookie (PULSAR_EXTRA_OPTS) [#10397](https://github.com/apache/pulsar/pull/10397) +- Allow the C++ client to be built in Windows and add CI for verification [#10387](https://github.com/apache/pulsar/pull/10387) +- Allow to activate every BookKeeper Client features in the Broker [#9232](https://github.com/apache/pulsar/pull/9232) +- Improvements Pulsar Proxy +- Upgrade core networking libraries: Jetty and Netty + +[Here](https://github.com/apache/pulsar/pulls?page=1=is%3Apr+label%3Arelease%2F2.7.2]) you can find the list of all the improvements and bugfixes. + +### Contributors for 2.7.2 release + +We would like to thank all the contributors for this release. +As every healthy Open Source Project, Apache Pulsar is great because it is supported by a vibrant community. + +Code contributors (names taken from GitHub API): +- Ali Ahmed +- Andrey Yegorov +- Binbin Guo +- David Kjerrumgaard +- Deon van der Vyver +- Devin Bost +- Enrico Olivelli +- Guangning E +- Kevin Wilson +- Lari Hotari +- Marvin Cai +- Masahiro Sakamoto +- Matteo Merli +- Michael Marshall +- Rajan Dhabalia +- Shen Liu +- Ting Yuan +- Vincent Royer +- Yong Zhang +- Yunze Xu +- Zhanpeng Wu +- Zike Yang +- baomingyu +- CongBo +- dockerzhang +- feynmanlin +- hangc0276 +- li jinquan +- limingnihao +- linlinnn +- mlyahmed +- PengHui Li +- ran + +Also, we want to thank everyone who spent his time to report issues and to share his story about using Pulsar. Review comment: ```suggestion Also, we want to thank everyone who spent his time reporting issues and sharing the story about using Pulsar. ``` Use gender-neutral language to avoid gender bias. ## File path: site2/website/blog/2021-05-13-Apache-Pulsar-2-7-2.md ## @@ -0,0 +1,68 @@ +--- +author: Enrico Olivelli +authorURL: https://twitter.com/eolivelli +title: Apache Pulsar 2.7.2 +--- +We are very glad to see the Apache Pulsar community has successfully released 2.7.2 version. +This is a minor release that introduces stability fixes and a few new features without breaking changes. + + + +### News and noteworthy + +Here is a selection of the most interesting and major features added to Pulsar 2.7.2. + +- Improvements in stability in the Kinesis Connector [#10420](https://github.com/apache/pulsar/pull/10420) +- Improvement in passing ENV variables to the Bookie (PULSAR_EXTRA_OPTS) [#10397](https://github.com/apache/pulsar/pull/10397) +- Allow the C++ client to be built in Windows and add CI for verification [#10387](https://github.com/apache/pulsar/pull/10387) +- Allow to activate every BookKeeper Client features in the Broker [#9232](https://github.com/apache/pulsar/pull/9232) +- Improvements Pulsar Proxy +- Upgrade core networking libraries: Jetty and Netty + +[Here](https://github.com/apache/pulsar/pulls?page=1=is%3Apr+label%3Arelease%2F2.7.2]) you can find the list of all the improvements and bugfixes. + +### Contributors for 2.7.2 release + +We would like to thank all the contributors for this release. +As every healthy Open Source Project, Apache Pulsar is great because it is supported by a vibrant community. + +Code contributors (names taken from GitHub API): +- Ali Ahmed +- Andrey Yegorov +- Binbin Guo +- David Kjerrumgaard +- Deon van der Vyver +- Devin Bost +- Enrico Olivelli +- Guangning E +- Kevin Wilson +- Lari Hotari +- Marvin Cai +- Masahiro Sakamoto +- Matteo Merli +- Michael Marshall +- Rajan Dhabalia +- Shen Liu +- Ting Yuan +- Vincent Royer +- Yong Zhang +- Yunze Xu +- Zhanpeng Wu +- Zike Yang +- baomingyu +- CongBo +- dockerzhang +- feynmanlin +- hangc0276 +- li jinquan +- limingnihao +- linlinnn +- mlyahmed +- PengHui Li +- ran Review comment: Better to use colons to separate names to take up less space. `Ali Ahmed, Andrey Yegorov, Binbin Guo, ...` ## File path: site2/website/blog/2021-05-13-Apache-Pulsar-2-7-2.md ## @@ -0,0 +1,68 @@ +--- +author: Enrico Olivelli +authorURL: https://twitter.com/eolivelli +title: Apache Pulsar 2.7.2 +--- +We are very glad to see the Apache Pulsar community has successfully released 2.7.2 version. +This is a minor release that introduces stability fixes and a
[GitHub] [pulsar] eolivelli commented on pull request #10384: [Broker] Fix various issues when using preciseTopicPublishRateLimiterEnable=true
eolivelli commented on pull request #10384: URL: https://github.com/apache/pulsar/pull/10384#issuecomment-841094967 @codelipenghui @merlimat @rdhabalia do you have any comment ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10586: Fix NPE when ACK grouping tracker checks duplicated message id
eolivelli commented on a change in pull request #10586:
URL: https://github.com/apache/pulsar/pull/10586#discussion_r632365085
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
##
@@ -210,7 +210,8 @@ public int compareTo(MessageId o) {
} else if (o instanceof TopicMessageIdImpl) {
return compareTo(((TopicMessageIdImpl) o).getInnerMessageId());
} else {
-throw new UnsupportedOperationException("Unknown MessageId type: "
+ o.getClass().getName());
+final String typeName = (o != null) ? o.getClass().getName() :
"null";
Review comment:
what about adding a check in the beginning of the method ? the code will
be easier to read. null checks are not expensive
```
if (o == null) {
throw new UnsupportedOperationException()
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10586: Fix NPE when ACK grouping tracker checks duplicated message id
eolivelli commented on a change in pull request #10586:
URL: https://github.com/apache/pulsar/pull/10586#discussion_r632365085
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
##
@@ -210,7 +210,8 @@ public int compareTo(MessageId o) {
} else if (o instanceof TopicMessageIdImpl) {
return compareTo(((TopicMessageIdImpl) o).getInnerMessageId());
} else {
-throw new UnsupportedOperationException("Unknown MessageId type: "
+ o.getClass().getName());
+final String typeName = (o != null) ? o.getClass().getName() :
"null";
Review comment:
we can add a check in the beginning of the method.
```
if (o == null) {
throw new UnsupportedOperationException()
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #10573: [Schema] Add schemaType field in SchemaHash
eolivelli commented on pull request #10573: URL: https://github.com/apache/pulsar/pull/10573#issuecomment-841092101 @gaoran10 can you please take a look to integration tests? I have never seen this failure before, it is probably not a flaky test but a problem with this patch  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #10540: Apache Pulsar 2.7.2 blog page
eolivelli commented on pull request #10540: URL: https://github.com/apache/pulsar/pull/10540#issuecomment-841086039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated (79757ac -> 959a8c8)
This is an automated email from the ASF dual-hosted git repository. eolivelli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 79757ac Update vcpkg.json package list (#10581) add 959a8c8 Show 2.7.2 release on WebSite (#10585) No new revisions were added by this update. Summary of changes: site2/website/releases.json | 1 + site2/website/versions.json | 1 + 2 files changed, 2 insertions(+)
[GitHub] [pulsar] eolivelli merged pull request #10585: Show 2.7.2 release on WebSite
eolivelli merged pull request #10585: URL: https://github.com/apache/pulsar/pull/10585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] atezs82 commented on pull request #10574: [Issue 905][api][pulsar-client] add serializable token class
atezs82 commented on pull request #10574: URL: https://github.com/apache/pulsar/pull/10574#issuecomment-841080238 @merlimat thanks for the review, I'll try to modify this accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on issue #10584: broker.conf documentation : wrong tokenSecretKey base64 inline description
Anonymitaet commented on issue #10584: URL: https://github.com/apache/pulsar/issues/10584#issuecomment-841079005 Hi @daniel-bartz have you tried `tokenSecretKey=data:;base64,**`? (with the semicolon) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower opened a new pull request #10586: Fix NPE when ACK grouping tracker checks duplicated message id
BewareMyPower opened a new pull request #10586: URL: https://github.com/apache/pulsar/pull/10586 ### Motivation Recently I encountered the following NPE but it's hard to reproduce. ``` 01:40:14.630 [pulsar-client-io-54-1] WARN org.apache.pulsar.client.impl.ClientCnx - [10.0.0.44/10.0.0.44:6650] Got exception java.lang.NullPointerException at org.apache.pulsar.client.impl.MessageIdImpl.compareTo(MessageIdImpl.java:213) at org.apache.pulsar.client.impl.MessageIdImpl.compareTo(MessageIdImpl.java:37) at org.apache.pulsar.client.impl.PersistentAcknowledgmentsGroupingTracker.isDuplicate(PersistentAcknowledgmentsGroupingTracker.java:117) at org.apache.pulsar.client.impl.ConsumerImpl.messageReceived(ConsumerImpl.java:1013) ``` From the stack we can see NPE was thrown when an ACK grouping tracker checks duplicated message id. The track maintains a `LastCumulativeAck` field that has a `MessageIdImpl` type field `messageId`. However, `messageId` could be null after `recycle()` or just was created with a null `MessageIdImpl`. We should check null here. Anyway, it may be introduced from https://github.com/apache/pulsar/pull/9440 ### Modifications - Check null in `PersistentAcknowledgmentsGroupingTracker#isDuplicate` and returns false if it's null, then the tracker will do nothing in `ConsumerImpl#messageReceived`. - Check null in `MessageIdImpl#compareTo` and throw `UnsupportedOperationException` if the compared object is null. ### Verifying this change - [ ] Make sure that the change passes the CI checks. This change is a trivial rework / code cleanup without any test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #10384: [Broker] Fix various issues when using preciseTopicPublishRateLimiterEnable=true
lhotari commented on a change in pull request #10384:
URL: https://github.com/apache/pulsar/pull/10384#discussion_r632344868
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PrecisPublishLimiter.java
##
@@ -71,31 +80,63 @@ public void update(Policies policies, String clusterName) {
: null;
this.update(maxPublishRate);
}
+
public void update(PublishRate maxPublishRate) {
-if (maxPublishRate != null
Review comment:
I think that would make the logic a bit complicated without much gains.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #10384: [Broker] Fix various issues when using preciseTopicPublishRateLimiterEnable=true
lhotari commented on a change in pull request #10384:
URL: https://github.com/apache/pulsar/pull/10384#discussion_r632344627
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
##
@@ -425,6 +425,13 @@ public void removeProducer(Producer producer) {
replicators.forEach((cluster, replicator) ->
futures.add(replicator.disconnect()));
producers.values().forEach(producer ->
futures.add(producer.disconnect()));
+if (topicPublishRateLimiter instanceof AutoCloseable) {
Review comment:
yes, I guess it makes sense in this case since PublishRateLimiter is an
internal interface and it's fine to change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] baomingyu commented on pull request #10285: fix issue 10284 Occasional consumer stucked when restart consumer whit key_shared subscription type
baomingyu commented on pull request #10285: URL: https://github.com/apache/pulsar/pull/10285#issuecomment-841065333 @eolivellihas done，please review this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli opened a new pull request #10585: Show 2.7.2 release on WebSite
eolivelli opened a new pull request #10585: URL: https://github.com/apache/pulsar/pull/10585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
