[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #18491: [fix][client] Fix multi-topic consumer stuck after redeliver messages
BewareMyPower commented on code in PR #18491:
URL: https://github.com/apache/pulsar/pull/18491#discussion_r1024855178
##
pulsar-broker/src/test/java/org/apache/pulsar/client/impl/NegativeAcksTest.java:
##
@@ -439,4 +443,76 @@ public void run() {
Assert.assertEquals(count, 9);
Assert.assertEquals(0, datas.size());
}
+
+/**
+ * see https://github.com/apache/pulsar/pull/18491
+ */
+@Test
+public void testMultiTopicConsumerConcurrentRedeliverAndReceive() throws
Exception {
+final String topic = BrokerTestUtil.newUniqueName("my-topic");
+admin.topics().createPartitionedTopic(topic, 2);
+
+final int receiverQueueSize = 10;
+
+@Cleanup
+MultiTopicsConsumerImpl consumer =
+(MultiTopicsConsumerImpl)
pulsarClient.newConsumer(Schema.INT32)
+.topic(topic)
+.subscriptionName("sub")
+.receiverQueueSize(receiverQueueSize)
+.subscribe();
+
+@Cleanup
+Producer producer = pulsarClient.newProducer(Schema.INT32)
+.topic(topic)
+.enableBatching(false)
+.create();
+
+for (int i = 0; i < receiverQueueSize; i++){
+producer.send(i);
+}
+
+Awaitility.await().until(() -> consumer.incomingMessages.size() ==
receiverQueueSize);
+
+// For testing the race condition of issue #18491
+// We need to inject a delay for the pinned internal thread
+injectDelayToInternalThread(consumer, 1000L);
+consumer.redeliverUnacknowledgedMessages();
+// Make sure the message redelivery is completed. The incoming queue
will be cleaned up during the redelivery.
+waitForAllTasksForInternalThread(consumer);
+
+Set receivedMsgs = new HashSet<>();
+for (;;){
+Message msg = consumer.receive(2, TimeUnit.SECONDS);
+if (msg == null){
+break;
+}
+receivedMsgs.add(msg.getValue());
+}
+Assert.assertEquals(receivedMsgs.size(), 10);
+}
+
+private void injectDelayToInternalThread(MultiTopicsConsumerImpl
consumer, long delayInMillis){
+ExecutorService internalPinnedExecutor =
+WhiteboxImpl.getInternalState(consumer,
"internalPinnedExecutor");
+internalPinnedExecutor.execute(() -> {
+try {
+Thread.sleep(delayInMillis);
+} catch (InterruptedException ignore) {
+}
+});
+}
+
+/**
+ * If the task after "redeliver" finish, means task-redeliver finish.
+ */
+private void waitForAllTasksForInternalThread(MultiTopicsConsumerImpl
consumer){
Review Comment:
```suggestion
private void waitForAllTasksForInternalThread(MultiTopicsConsumerImpl
consumer) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] labuladong commented on issue #18515: [PulsarStandalone]The PulsarStandalon mode allows you to configure multiple brokers and multiple bookie launches.
labuladong commented on issue #18515: URL: https://github.com/apache/pulsar/issues/18515#issuecomment-1318230699 For now standalone can start multiple bookies with `--num-bookies`, try this for more information: ```shell ./bin/pulsar standalone --help ``` And I think this is a good suggestion to explore Pulsar locally if we can start multiple brokers. But seems it's not supported now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.9 updated: [enh][broker] Add metrics for entry cache insertion, eviction (#17248)
This is an automated email from the ASF dual-hosted git repository.
bogong pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new bd0c56a3145 [enh][broker] Add metrics for entry cache insertion,
eviction (#17248)
bd0c56a3145 is described below
commit bd0c56a31459e1e4a83f34e7a11c209c8faf7dc0
Author: Michael Marshall
AuthorDate: Wed Aug 24 14:00:45 2022 -0700
[enh][broker] Add metrics for entry cache insertion, eviction (#17248)
Fixes https://github.com/apache/pulsar/issues/16584
With the `RangeCache`, it is hard to reason about its behavior other than
cache hits/misses or the cache's size hitting the limit and triggering a size
based eviction. This PR adds 3 new metrics to help provide additional insight
into the cache's behavior. It adds `pulsar_ml_cache_inserted_entries_total`,
`pulsar_ml_cache_evicted_entries_total`, and `pulsar_ml_cache_entries`.
* Add new metrics for cache insertion, eviction, and current number of
entries.
* Add new methods to the `ManagedLedgerFactoryMXBean` interface.
* Update several method return values in the `RangeCache`.
* Update tests.
This change is covered by modified tests that already existed.
There is a breaking change to the `RangeCache` class for the `clear` and
the `evictLEntriesBeforeTimestamp` methods. The previous result was a `long`,
and now it is a `Pair`. The new result matches the same style as
`evictLeastAccessedEntries`. Given that this class is only meant for use within
the broker, I think it is reasonable to break these methods. I will send a note
to the mailing list.
- [x] `doc`
(cherry picked from commit e3b25403e4abb6a0c16073e5fb28a533497d094c)
---
.../mledger/ManagedLedgerFactoryMXBean.java| 15
.../impl/ManagedLedgerFactoryMBeanImpl.java| 27 ++
.../mledger/impl/cache/RangeEntryCacheImpl.java| 14
.../impl/cache/RangeEntryCacheManagerImpl.java | 4 ++-
.../apache/bookkeeper/mledger/util/RangeCache.java | 12 ---
.../mledger/impl/EntryCacheManagerTest.java| 18 ++
.../bookkeeper/mledger/util/RangeCacheTest.java| 5 +--
.../stats/metrics/ManagedLedgerCacheMetrics.java | 3 ++
site2/docs/reference-metrics.md| 41 ++
9 files changed, 125 insertions(+), 14 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryMXBean.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryMXBean.java
index ea5b2074ffa..f71583ab886 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryMXBean.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryMXBean.java
@@ -66,4 +66,19 @@ public interface ManagedLedgerFactoryMXBean {
* Get the number of cache evictions during the last minute.
*/
long getNumberOfCacheEvictions();
+
+/**
+ * Cumulative number of entries inserted into the cache.
+ */
+long getCacheInsertedEntriesCount();
+
+/**
+ * Cumulative number of entries evicted from the cache.
+ */
+long getCacheEvictedEntriesCount();
+
+/**
+ * Current number of entries in the cache.
+ */
+long getCacheEntriesCount();
}
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java
index d514d8381d9..a5f0c67e68a 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java
@@ -19,6 +19,7 @@
package org.apache.bookkeeper.mledger.impl;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.LongAdder;
import org.apache.bookkeeper.mledger.ManagedLedgerFactoryMXBean;
import org.apache.pulsar.common.stats.Rate;
@@ -31,6 +32,10 @@ public class ManagedLedgerFactoryMBeanImpl implements
ManagedLedgerFactoryMXBean
final Rate cacheMisses = new Rate();
final Rate cacheEvictions = new Rate();
+private final LongAdder insertedEntryCount = new LongAdder();
+private final LongAdder evictedEntryCount = new LongAdder();
+private final LongAdder cacheEntryCount = new LongAdder();
+
public ManagedLedgerFactoryMBeanImpl(ManagedLedgerFactoryImpl factory)
throws Exception {
this.factory = factory;
}
@@ -64,6 +69,16 @@ public class ManagedLedgerFactoryMBeanImpl implements
ManagedLedgerFactoryMXBean
cacheEvictions.recordEvent();
}
+public void recordCacheInsertion() {
+insertedEntryCount.increment();
+cacheEntryCount.increment();
+}
[GitHub] [pulsar] BewareMyPower commented on pull request #18491: [fix][client] Fix multi-topic consumer stuck after redeliver messages
BewareMyPower commented on PR #18491:
URL: https://github.com/apache/pulsar/pull/18491#issuecomment-1318224636
Got it. Thanks. The motivation part is a little confusing. The root cause is
that the consumer won't be removed from `pausedConsumers` if the incoming
messages are greater than half of the receiver queue size. Therefore,
`resumeReceivingFromPausedConsumersIfNeeded` must be called after
`clearIncomingMessages`.
BTW, it's not a race condition in `redeliverUnacknowledgedMessages` for
previous code:
```java
public void redeliverUnacknowledgedMessages() {
internalPinnedExecutor.execute(() -> {
/* ... */
clearIncomingMessages();
unAckedMessageTracker.clear();
});
resumeReceivingFromPausedConsumersIfNeeded();
}
```
`internalPinnedExecutor` is a single thread executor, when
`redeliverUnacknowledgedMessages` is called in `internalPinnedExecutor`,
`resumeReceivingFromPausedConsumersIfNeeded` will always be called before
`clearIncomingMessages()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar] branch branch-2.9 updated: Extracted interface for EntryCacheManager (#15933)
This is an automated email from the ASF dual-hosted git repository.
bogong pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new ee5f5d251c4 Extracted interface for EntryCacheManager (#15933)
ee5f5d251c4 is described below
commit ee5f5d251c4ac1645d8c6641a6902714d2c6b45b
Author: Matteo Merli
AuthorDate: Mon Jun 6 16:39:14 2022 -0700
Extracted interface for EntryCacheManager (#15933)
* Extracted interface for EntryCacheManager
* Fixed references
* added more methods to the interface
* Fixed mocked test
* Removed unused import
* Fixed wrong casting in reflection access
(cherry picked from commit c7faf6248b1413fc71465fc12571886db323193c)
---
.../bookkeeper/mledger/ManagedLedgerFactory.java | 2 +-
.../bookkeeper/mledger/impl/ManagedCursorImpl.java | 4 +-
.../mledger/impl/ManagedLedgerFactoryImpl.java | 6 +-
.../bookkeeper/mledger/impl/ManagedLedgerImpl.java | 25 ++-
.../mledger/impl/{ => cache}/EntryCache.java | 4 +-
.../EntryCacheDefaultEvictionPolicy.java | 4 +-
.../mledger/impl/cache/EntryCacheDisabled.java | 147 ++
.../impl/{ => cache}/EntryCacheEvictionPolicy.java | 2 +-
.../EntryCacheManager.java}| 34 ++--
.../RangeEntryCacheImpl.java} | 21 +-
.../RangeEntryCacheManagerImpl.java} | 144 ++
.../package-info.java} | 20 +-
.../mledger/impl/EntryCacheManagerTest.java| 126 ++--
.../bookkeeper/mledger/impl/EntryCacheTest.java| 4 +-
.../mledger/impl/ManagedLedgerBkTest.java | 11 +-
.../bookkeeper/mledger/impl/ManagedLedgerTest.java | 212 ++---
.../service/nonpersistent/NonPersistentTopic.java | 3 +-
.../broker/stats/AllocatorStatsGenerator.java | 4 +-
.../stats/metrics/ManagedLedgerCacheMetrics.java | 4 +-
.../broker/service/PersistentTopicE2ETest.java | 6 +-
.../client/api/SimpleProducerConsumerTest.java | 14 +-
.../client/api/v1/V1_ProducerConsumerTest.java | 15 +-
22 files changed, 420 insertions(+), 392 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactory.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactory.java
index a83994d1cd3..27ee3d4cb44 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactory.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactory.java
@@ -26,7 +26,7 @@ import
org.apache.bookkeeper.mledger.AsyncCallbacks.DeleteLedgerCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.ManagedLedgerInfoCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.OpenLedgerCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.OpenReadOnlyCursorCallback;
-import org.apache.bookkeeper.mledger.impl.EntryCacheManager;
+import org.apache.bookkeeper.mledger.impl.cache.EntryCacheManager;
/**
* A factory to open/create managed ledgers and delete them.
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
index 51b6ebaabab..ab97c10b687 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
@@ -2766,8 +2766,8 @@ public class ManagedCursorImpl implements ManagedCursor {
boolean shouldCloseLedger(LedgerHandle lh) {
long now = clock.millis();
-if (ledger.factory.isMetadataServiceAvailable() &&
-(lh.getLastAddConfirmed() >=
config.getMetadataMaxEntriesPerLedger()
+if (ledger.getFactory().isMetadataServiceAvailable()
+&& (lh.getLastAddConfirmed() >=
config.getMetadataMaxEntriesPerLedger()
|| lastLedgerSwitchTimestamp < (now -
config.getLedgerRolloverTimeout() * 1000))
&& (STATE_UPDATER.get(this) != State.Closed &&
STATE_UPDATER.get(this) != State.Closing)) {
// It's safe to modify the timestamp since this method will be
only called from a callback, implying that
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
index c3e391ffab2..dce1789cd6c 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
@@ -67,6 +67,8 @@ import org.apache.bookkeeper.mledger.ReadOnlyCursor;
import
[GitHub] [pulsar] mingyifei opened a new issue, #18515: [PulsarStandalone]The PulsarStandalon mode allows you to configure multiple brokers and multiple bookie launches.
mingyifei opened a new issue, #18515: URL: https://github.com/apache/pulsar/issues/18515 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Motivation To facilitate testing of broker functions, such as topic ownership transfer, simulating outages in extreme scenarios, etc. ### Solution _No response_ ### Alternatives _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] labuladong opened a new issue, #18514: [Bug] `pulsar-admin namespaces set-retention` not work
labuladong opened a new issue, #18514: URL: https://github.com/apache/pulsar/issues/18514 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version Master version of Pulsar. ### Minimal reproduce step 1. start stanalone Pulsar. 2. use `pulsar-admin namespace set-retention` then `pulsar-admin namespace get-retention`:  ### What did you expect to see? Retention setting has changed. ### What did you see instead? Retention setting didn't change. ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.9 updated: [pulsar-broker] Broker extensions to allow operators of enterprise wide cluster better control and flexibility (#12536)
This is an automated email from the ASF dual-hosted git repository.
bogong pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new 887e4d99838 [pulsar-broker] Broker extensions to allow operators of
enterprise wide cluster better control and flexibility (#12536)
887e4d99838 is described below
commit 887e4d99838b986525ba36bc27312bf2dca00aa7
Author: madhavan-narayanan
<92708902+madhavan-naraya...@users.noreply.github.com>
AuthorDate: Tue Dec 14 12:19:07 2021 +0530
[pulsar-broker] Broker extensions to allow operators of enterprise wide
cluster better control and flexibility (#12536)
Operators of enterprise Pulsar cluster(s) would need the flexibility and
control to intercept broker events (including ledger writes/reads) for template
validations, observability and access control. This changeset contains
enhancements to existing interceptor mechanism to support this
- Enhanced org.apache.pulsar.broker.intercept.BrokerInterceptor interface
to include additional events for tracing
- Created a new interface
org.apache.pulsar.common.intercept.MessagePayloadProcessor to allow
interception of ledger write/read operations
- Enhanced PulsarAdmin to give operators a control in managing super-users
- [x ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- *Added new test cases to MangedLedgerInterceptorImplTest.java and
BrokerInterceptorTest.java *
(cherry picked from commit 03bbc8e67ad429a774936d26ab2f4d40c1ea4999)
---
.../bookkeeper/mledger/impl/EntryCacheImpl.java| 7 +-
.../bookkeeper/mledger/impl/EntryCacheManager.java | 29 ++-
.../bookkeeper/mledger/impl/ManagedLedgerImpl.java | 6 +-
.../apache/bookkeeper/mledger/impl/OpAddEntry.java | 20 +
.../intercept/ManagedLedgerInterceptor.java| 35 +
.../apache/pulsar/broker/ServiceConfiguration.java | 6 ++
.../pulsar/broker/intercept/BrokerInterceptor.java | 65
.../BrokerInterceptorWithClassLoader.java | 43 ++
.../broker/intercept/BrokerInterceptors.java | 74 ++
.../intercept/ManagedLedgerInterceptorImpl.java| 63 ++-
.../pulsar/broker/service/BrokerService.java | 15 +++-
.../org/apache/pulsar/broker/service/Consumer.java | 4 +
.../org/apache/pulsar/broker/service/Producer.java | 36 +
.../apache/pulsar/broker/service/ServerCnx.java| 27 ++-
.../org/apache/pulsar/broker/service/Topic.java| 7 ++
.../broker/intercept/BrokerInterceptorTest.java| 32
.../broker/intercept/CounterBrokerInterceptor.java | 91 ++
.../intercept/MangedLedgerInterceptorImplTest.java | 80 +--
.../common/intercept/BrokerEntryMetadataUtils.java | 27 ++-
.../intercept/ManagedLedgerPayloadProcessor.java | 62 +++
20 files changed, 710 insertions(+), 19 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
index 085923e25cc..0851d942bd1 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java
@@ -38,6 +38,7 @@ import org.apache.bookkeeper.client.api.ReadHandle;
import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback;
import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.intercept.ManagedLedgerInterceptor;
import org.apache.bookkeeper.mledger.util.RangeCache;
import org.apache.commons.lang3.tuple.Pair;
import org.slf4j.Logger;
@@ -50,6 +51,7 @@ public class EntryCacheImpl implements EntryCache {
private final EntryCacheManager manager;
private final ManagedLedgerImpl ml;
+private ManagedLedgerInterceptor interceptor;
private final RangeCache entries;
private final boolean copyEntries;
@@ -58,6 +60,7 @@ public class EntryCacheImpl implements EntryCache {
public EntryCacheImpl(EntryCacheManager manager, ManagedLedgerImpl ml,
boolean copyEntries) {
this.manager = manager;
this.ml = ml;
+this.interceptor = ml.getManagedLedgerInterceptor();
this.entries = new RangeCache<>(EntryImpl::getLength,
EntryImpl::getTimestamp);
this.copyEntries = copyEntries;
@@ -215,7 +218,7 @@ public class EntryCacheImpl implements EntryCache {
Iterator iterator =
ledgerEntries.iterator();
if (iterator.hasNext()) {
LedgerEntry ledgerEntry = iterator.next();
-
[GitHub] [pulsar] coderzc closed pull request #18499: [improve][client] Avoid `SingleThreadExecutor` submit tasks recursively
coderzc closed pull request #18499: [improve][client] Avoid `SingleThreadExecutor` submit tasks recursively URL: https://github.com/apache/pulsar/pull/18499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] coderzc commented on pull request #18499: [improve][client] Avoid `SingleThreadExecutor` submit tasks recursively
coderzc commented on PR #18499: URL: https://github.com/apache/pulsar/pull/18499#issuecomment-1318185938 > Generally, we should avoid any blocking operations in Pulsar's core code. This executor encourage users to write synchronous code if they have imported the Pulsar dependency. If an external system adopted this new executor and executed a blocking operation. Then once we removed this executor in future, the dead lock could occur in that external system. Agree, to avoid the users confusion, I decided to close the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #18499: [improve][client] Avoid `SingleThreadExecutor` submit tasks recursively
BewareMyPower commented on PR #18499: URL: https://github.com/apache/pulsar/pull/18499#issuecomment-1318178628 Generally, we should avoid any blocking operations in Pulsar's core code. This executor encourage users to write synchronous code if they have imported the Pulsar dependency. If an external system adopted this new executor and executed a blocking operation. Then once we removed this executor in future, the dead lock could occur in that external system. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] coderzc commented on pull request #18499: [improve][client] Avoid `SingleThreadExecutor` submit tasks recursively
coderzc commented on PR #18499:
URL: https://github.com/apache/pulsar/pull/18499#issuecomment-1318153620
> The code above works well with the following outputs:
You can try the following code, and will not be able to work well:
```java
final var executor = Executors.newSingleThreadExecutor();
executor.execute(() -> {
final var latch = new CountDownLatch(1);
executor.execute(() -> {
System.out.println(Thread.currentThread().getName() + " 1");
latch.countDown();
});
latch.await();
System.out.println(Thread.currentThread().getName() + " 0");
});
executor.shutdown();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18418: [fix][test] Fix ConcurrentModificationException in testConcurrentWriteBrokerData
codecov-commenter commented on PR #18418: URL: https://github.com/apache/pulsar/pull/18418#issuecomment-1318153213 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18418?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#18418](https://codecov.io/gh/apache/pulsar/pull/18418?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (e1ffdeb) into [master](https://codecov.io/gh/apache/pulsar/commit/27186a1ee9298802a12f1d42f9bba03d8f3eea77?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (27186a1) will **decrease** coverage by `8.80%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/pulsar/pull/18418?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #18418 +/- ## - Coverage 45.61% 36.81% -8.81% + Complexity10728 1949-8779 Files 752 208 -544 Lines 7252114426 -58095 Branches 7791 1582-6209 - Hits 33083 5311 -27772 + Misses35769 8522 -27247 + Partials 3669 593-3076 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `36.81% <ø> (-8.81%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18418?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...pache/pulsar/client/impl/schema/ByteBufSchema.java](https://codecov.io/gh/apache/pulsar/pull/18418/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL3NjaGVtYS9CeXRlQnVmU2NoZW1hLmphdmE=) | `68.75% <0.00%> (-31.25%)` | :arrow_down: | | [...pulsar/client/impl/schema/LocalDateTimeSchema.java](https://codecov.io/gh/apache/pulsar/pull/18418/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL3NjaGVtYS9Mb2NhbERhdGVUaW1lU2NoZW1hLmphdmE=) | `76.92% <0.00%> (-23.08%)` | :arrow_down: | | [...g/apache/pulsar/client/impl/schema/ByteSchema.java](https://codecov.io/gh/apache/pulsar/pull/18418/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL3NjaGVtYS9CeXRlU2NoZW1hLmphdmE=) | `50.00% <0.00%> (-19.24%)` | :arrow_down: | | [...he/pulsar/client/impl/schema/ByteBufferSchema.java](https://codecov.io/gh/apache/pulsar/pull/18418/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL3NjaGVtYS9CeXRlQnVmZmVyU2NoZW1hLmphdmE=) | `43.33% <0.00%> (-16.67%)` | :arrow_down: | | [...apache/pulsar/client/impl/schema/StringSchema.java](https://codecov.io/gh/apache/pulsar/pull/18418/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL3NjaGVtYS9TdHJpbmdTY2hlbWEuamF2YQ==) | `78.26% <0.00%> (-13.05%)` | :arrow_down: | | [.../apache/pulsar/client/impl/schema/BytesSchema.java](https://codecov.io/gh/apache/pulsar/pull/18418/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL3NjaGVtYS9CeXRlc1NjaGVtYS5qYXZh) | `88.23% <0.00%> (-11.77%)` | :arrow_down: | |
[GitHub] [pulsar] RobertIndie commented on pull request #18505: [refactor][broker] Optimize semantics
RobertIndie commented on PR #18505: URL: https://github.com/apache/pulsar/pull/18505#issuecomment-1318140756 @crossoverJie Seems that you haven't enabled the GitHub action in your repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #18416: [fix][broker] Fix open cursor with null-initialPosition result with earliest position
AnonHxy commented on PR #18416: URL: https://github.com/apache/pulsar/pull/18416#issuecomment-1318140712 @aloyszhang @Technoboy- PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] labuladong opened a new pull request, #18512: [fix][broker] namespace not found will cause request timeout
labuladong opened a new pull request, #18512: URL: https://github.com/apache/pulsar/pull/18512 Fixes #18497 ### Modifications `internalGetSubscriptions` didn't handle `namespace not found` exception so the web request won't get response, which causes request timout. ### Verifying this change - [x] Make sure that the change passes the CI checks. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: https://github.com/labuladong/pulsar/pull/13 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #18418: [fix][test] Fix ConcurrentModificationException in testConcurrentWriteBrokerData
AnonHxy commented on PR #18418: URL: https://github.com/apache/pulsar/pull/18418#issuecomment-1318139436 Updated. PTAL @poorbarcode -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] RobertIndie opened a new pull request, #18511: [fix][client] Avoid allocating unused buffer when receiving chunk message
RobertIndie opened a new pull request, #18511: URL: https://github.com/apache/pulsar/pull/18511 ### Motivation Currently, if there are duplicated messages whose chunk id is both 0, then it may result in allocating an unused buffer and may lead to the buffer memory leak. ### Modifications * Only allocate the bytebuffer when there is no duplicated chunk message with chunk id 0. ### Verifying this change This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] pkumar-singh opened a new issue, #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
pkumar-singh opened a new issue, #18510: URL: https://github.com/apache/pulsar/issues/18510 ### Motivation When a topic is a partitioned topic and a partition is not available for producing messages, currently pulsar client will still try to produce messages on unavailable partitions, which it may not necessarily need to do in certain cases. Pulsar Client may simply pick up another partition and try producing in certain cases. Partition Unavailable There could be a plethora of reasons a partition can become unavailable. But the most prominent reason is partition is moving from one broker to another, and until every actor is in sync with which broker owns the partition, the partition will be unavailable for producing. Actors are producers, old broker, new broker. ### Goal Produce uninterrupted as long as possible when a partition is down. ### API Changes pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java /** * This config will ensure that If possible PartitionedProducer would attempt to produce message on * another available partitions, If currently picked partition is not available for some reason. * Next available partition will be chosen by the same routing policy as client is configured with. * @param maxRetryOtherPartition *How many partitions should be tried before bailing out * @return the producer builder instance */ ProducerBuilder maxRetryOtherPartitions(int maxRetryOtherPartition); ### Implementation **Client Behavior** This is the typical produce code. producer.sendAsync(payLoad.getBytes(StandardCharsets.UTF_8)); When send is called message is enqueued in a queue(called pending message queue) and the future is returned. And future is only completed when the message is picked from the queue and sent to the broker asynchronously and ack is received asynchronously again. Max size of the pending message queue is controlled by producer config maxPendingMessages. When pending message queue is full, the application will start getting publish failures. Pending message queue provide a cushion towards unavailable partitions. But again it has some limits. **When another partitions can be picked** When the message is not keyed. That means the message is not ordered based on a key. When routing mode is round-robin, that means a message can be produced to any of the partitions. So If a partition is unavailable try and pick up another partition for producing, by using the same round-robin algorithm. ### Alternatives _No response_ ### Anything else? My suggestion is to keep Router(RoundRobin) not dependent on whether a partition is available or not. Or batching is enabled or publish is happening under transaction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#6f6e32e)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 756e89b135c Docs sync done from apache/pulsar(#6f6e32e) 756e89b135c is described below commit 756e89b135c0d2abf7210eb67ed6d6aab9253a9f Author: Pulsar Site Updater AuthorDate: Thu Nov 17 06:01:56 2022 + Docs sync done from apache/pulsar(#6f6e32e)
[GitHub] [pulsar] poorbarcode commented on pull request #18418: [fix][broker] Fix ConcurrentModificationException in LocalBrokerData
poorbarcode commented on PR #18418: URL: https://github.com/apache/pulsar/pull/18418#issuecomment-1318059846 Hi @AnonHxy > It make sense to me. It seems that we could just remove the following from the test case, and keep others unchanged ``` LocalBrokerData data = loadManager.getLoadManager().updateLocalBrokerData(); data.cleanDeltas(); data.getBundles().clear(); ``` > WDYT @poorbarcode I don't know why `data.clear` should executed in this place (-_-) > @poorbarcode do you have any other comments？:) I think the concurrent operation was caused by the test case, so there is no need to change the collection to be thread-safe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] crossoverJie commented on pull request #18505: [refactor][broker] Optimize semantics
crossoverJie commented on PR #18505: URL: https://github.com/apache/pulsar/pull/18505#issuecomment-1318037562 > @crossoverJie Please follow the instructions in https://github.com/apache/pulsar/actions/runs/3481081589/attempts/1#summary-9529786678 . Done, PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] crossoverJie commented on pull request #18505: [refactor][broker] Optimize semantics
crossoverJie commented on PR #18505: URL: https://github.com/apache/pulsar/pull/18505#issuecomment-1318021736 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] crossoverJie commented on pull request #18505: [refactor][broker] Optimize semantics
crossoverJie commented on PR #18505: URL: https://github.com/apache/pulsar/pull/18505#issuecomment-1318021309 PR in forked repository: https://github.com/crossoverJie/pulsar/pull/2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
svn commit: r58063 - in /dev/pulsar/pulsar-2.11.0-candidate-1: ./ RPMS/ connectors/
Author: technoboy Date: Thu Nov 17 03:34:52 2022 New Revision: 58063 Log: Staging artifacts and signature for Pulsar release 2.11.0 Added: dev/pulsar/pulsar-2.11.0-candidate-1/ dev/pulsar/pulsar-2.11.0-candidate-1/RPMS/ dev/pulsar/pulsar-2.11.0-candidate-1/apache-pulsar-2.11.0-bin.tar.gz (with props) dev/pulsar/pulsar-2.11.0-candidate-1/apache-pulsar-2.11.0-src.tar.gz (with props) dev/pulsar/pulsar-2.11.0-candidate-1/apache-pulsar-offloaders-2.11.0-bin.tar.gz (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/ dev/pulsar/pulsar-2.11.0-candidate-1/connectors/LICENSE dev/pulsar/pulsar-2.11.0-candidate-1/connectors/README dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-aerospike-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-batch-data-generator-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-canal-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-cassandra-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-data-generator-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-debezium-mongodb-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-debezium-mssql-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-debezium-mysql-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-debezium-oracle-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-debezium-postgres-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-dynamodb-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-elastic-search-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-file-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-flume-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-hbase-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-hdfs2-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-hdfs3-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-http-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-influxdb-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-jdbc-clickhouse-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-jdbc-mariadb-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-jdbc-openmldb-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-jdbc-postgres-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-jdbc-sqlite-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-kafka-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-kafka-connect-adaptor-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-kinesis-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-mongo-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-netty-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-nsq-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-rabbitmq-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-redis-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-solr-2.11.0.nar (with props) dev/pulsar/pulsar-2.11.0-candidate-1/connectors/pulsar-io-twitter-2.11.0.nar (with props) Added: dev/pulsar/pulsar-2.11.0-candidate-1/apache-pulsar-2.11.0-bin.tar.gz == Binary file - no diff available. Propchange: dev/pulsar/pulsar-2.11.0-candidate-1/apache-pulsar-2.11.0-bin.tar.gz -- svn:mime-type = application/octet-stream Added: dev/pulsar/pulsar-2.11.0-candidate-1/apache-pulsar-2.11.0-src.tar.gz == Binary file - no diff available. Propchange: dev/pulsar/pulsar-2.11.0-candidate-1/apache-pulsar-2.11.0-src.tar.gz -- svn:mime-type = application/octet-stream Added: dev/pulsar/pulsar-2.11.0-candidate-1/apache-pulsar-offloaders-2.11.0-bin.tar.gz
[GitHub] [pulsar] gaozhangmin commented on issue #18507: [Bug] Pulsar Admin sink update can potentially wipe out "inputSpecs"
gaozhangmin commented on issue #18507: URL: https://github.com/apache/pulsar/issues/18507#issuecomment-1318014642 I will check this out -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.9 updated: Add a cache eviction policy：Evicting cache data by the slowest markDeletedPosition (#14985)
This is an automated email from the ASF dual-hosted git repository.
bogong pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new f31a6fae95e Add a cache eviction policy：Evicting cache data by the
slowest markDeletedPosition (#14985)
f31a6fae95e is described below
commit f31a6fae95e748a931afb48e2a471a199279eb2d
Author: LinChen <1572139...@qq.com>
AuthorDate: Wed Apr 6 15:53:58 2022 +0800
Add a cache eviction policy：Evicting cache data by the slowest
markDeletedPosition (#14985)
(cherry picked from commit 9b36dcd777b7e3b159b57cd409784a3622483132)
---
.../bookkeeper/mledger/ManagedLedgerConfig.java| 6 +
.../mledger/impl/ManagedCursorContainer.java | 4 +
.../bookkeeper/mledger/impl/ManagedLedgerImpl.java | 25 ++-
.../bookkeeper/mledger/impl/ManagedLedgerTest.java | 198 +
.../apache/pulsar/broker/ServiceConfiguration.java | 8 +-
.../pulsar/broker/service/BrokerService.java | 2 +
6 files changed, 238 insertions(+), 5 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java
index 2f4e098d677..41fc26c5099 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java
@@ -26,6 +26,8 @@ import java.util.Arrays;
import java.util.Map;
import java.util.concurrent.TimeUnit;
+import lombok.Getter;
+import lombok.Setter;
import org.apache.bookkeeper.client.EnsemblePlacementPolicy;
import org.apache.bookkeeper.client.api.DigestType;
@@ -78,6 +80,10 @@ public class ManagedLedgerConfig {
private Clock clock = Clock.systemUTC();
private ManagedLedgerInterceptor managedLedgerInterceptor;
+@Getter
+@Setter
+private boolean cacheEvictionByMarkDeletedPosition = false;
+
public boolean isCreateIfMissing() {
return createIfMissing;
}
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainer.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainer.java
index c631cdcf96d..4c3d4134379 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainer.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainer.java
@@ -108,6 +108,10 @@ public class ManagedCursorContainer implements
Iterable {
return heap.isEmpty() ? null : (PositionImpl)
heap.get(0).cursor.getReadPosition();
}
+public PositionImpl getSlowestMarkDeletedPositionForActiveCursors() {
+return heap.isEmpty() ? null : (PositionImpl)
heap.get(0).cursor.getMarkDeletedPosition();
+}
+
public ManagedCursor get(String name) {
long stamp = rwLock.readLock();
try {
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
index 72392d519c7..8a60a0b9936 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
@@ -2125,10 +2125,15 @@ public class ManagedLedgerImpl implements
ManagedLedger, CreateCallback {
if (entryCache.getSize() <= 0) {
return;
}
-// Always remove all entries already read by active cursors
-PositionImpl slowestReaderPos =
getEarlierReadPositionForActiveCursors();
-if (slowestReaderPos != null) {
-entryCache.invalidateEntries(slowestReaderPos);
+PositionImpl evictionPos;
+if (config.isCacheEvictionByMarkDeletedPosition()) {
+evictionPos =
getEarlierMarkDeletedPositionForActiveCursors().getNext();
+} else {
+// Always remove all entries already read by active cursors
+evictionPos = getEarlierReadPositionForActiveCursors();
+}
+if (evictionPos != null) {
+entryCache.invalidateEntries(evictionPos);
}
// Remove entries older than the cutoff threshold
@@ -2147,6 +2152,18 @@ public class ManagedLedgerImpl implements ManagedLedger,
CreateCallback {
return durablePosition.compareTo(nonDurablePosition) > 0 ?
nonDurablePosition : durablePosition;
}
+private PositionImpl getEarlierMarkDeletedPositionForActiveCursors() {
+PositionImpl nonDurablePosition =
nonDurableActiveCursors.getSlowestMarkDeletedPositionForActiveCursors();
+PositionImpl durablePosition =
activeCursors.getSlowestMarkDeletedPositionForActiveCursors();
+if (nonDurablePosition == null) {
+
[GitHub] [pulsar] nodece commented on pull request #18358: [fix][admin] Add support for admin auth via conf props
nodece commented on PR #18358: URL: https://github.com/apache/pulsar/pull/18358#issuecomment-1318007221 @cbornet Could you review this PR? Thank you for your time! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] StevenLuMT commented on pull request #18502: [improve][broker] Unreasonable AuthenticationException reference
StevenLuMT commented on PR #18502: URL: https://github.com/apache/pulsar/pull/18502#issuecomment-1317997821 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #18130: [fix][broker] Fix update authentication data
nodece commented on PR #18130: URL: https://github.com/apache/pulsar/pull/18130#issuecomment-1317995277 @codelipenghui @michaeljmarshall @Technoboy- Could you review this PR? Thanks for your time! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #18418: [fix][broker] Fix ConcurrentModificationException in LocalBrokerData
AnonHxy commented on PR #18418: URL: https://github.com/apache/pulsar/pull/18418#issuecomment-1317984026 @poorbarcode do you have any other comments？:) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.9 updated: Support dynamic update cache config (#13679)
This is an automated email from the ASF dual-hosted git repository.
bogong pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new 5093c07abf2 Support dynamic update cache config (#13679)
5093c07abf2 is described below
commit 5093c07abf22c4b0bb30dfae3d3965467588f8bd
Author: LinChen <1572139...@qq.com>
AuthorDate: Tue Mar 15 00:59:53 2022 +0800
Support dynamic update cache config (#13679)
(cherry picked from commit b0213b225f194b47a5dcd63ef4a26f55c4c820b6)
---
.../bookkeeper/mledger/ManagedLedgerFactory.java | 18 ++
.../bookkeeper/mledger/impl/EntryCacheManager.java | 19 ---
.../mledger/impl/ManagedLedgerFactoryImpl.java | 13 -
.../apache/pulsar/broker/ServiceConfiguration.java | 3 +++
.../pulsar/broker/service/BrokerService.java | 22 ++
.../apache/pulsar/broker/admin/AdminApiTest.java | 22 ++
6 files changed, 93 insertions(+), 4 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactory.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactory.java
index 682ce9008f1..a83994d1cd3 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactory.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactory.java
@@ -26,6 +26,7 @@ import
org.apache.bookkeeper.mledger.AsyncCallbacks.DeleteLedgerCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.ManagedLedgerInfoCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.OpenLedgerCallback;
import org.apache.bookkeeper.mledger.AsyncCallbacks.OpenReadOnlyCursorCallback;
+import org.apache.bookkeeper.mledger.impl.EntryCacheManager;
/**
* A factory to open/create managed ledgers and delete them.
@@ -179,4 +180,21 @@ public interface ManagedLedgerFactory {
*/
CompletableFuture asyncExists(String ledgerName);
+/**
+ * @return return EntryCacheManager.
+ */
+EntryCacheManager getEntryCacheManager();
+
+/**
+ * update cache evictionTimeThreshold.
+ *
+ * @param cacheEvictionTimeThresholdNanos time threshold for eviction.
+ */
+void updateCacheEvictionTimeThreshold(long
cacheEvictionTimeThresholdNanos);
+
+/**
+ * @return time threshold for eviction.
+ * */
+long getCacheEvictionTimeThreshold();
+
}
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheManager.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheManager.java
index 1c0c288e878..acd01129694 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheManager.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheManager.java
@@ -44,9 +44,9 @@ import org.slf4j.LoggerFactory;
@SuppressWarnings("checkstyle:javadoctype")
public class EntryCacheManager {
-private final long maxSize;
-private final long evictionTriggerThreshold;
-private final double cacheEvictionWatermark;
+private volatile long maxSize;
+private volatile long evictionTriggerThreshold;
+private volatile double cacheEvictionWatermark;
private final AtomicLong currentSize = new AtomicLong(0);
private final ConcurrentMap caches =
Maps.newConcurrentMap();
private final EntryCacheEvictionPolicy evictionPolicy;
@@ -87,6 +87,15 @@ public class EntryCacheManager {
}
}
+public void updateCacheSizeAndThreshold(long maxSize) {
+this.maxSize = maxSize;
+this.evictionTriggerThreshold = (long) (maxSize *
evictionTriggerThresholdPercent);
+}
+
+public void updateCacheEvictionWatermark(double cacheEvictionWatermark) {
+this.cacheEvictionWatermark = cacheEvictionWatermark;
+}
+
void removeEntryCache(String name) {
EntryCache entryCache = caches.remove(name);
if (entryCache == null) {
@@ -148,6 +157,10 @@ public class EntryCacheManager {
return maxSize;
}
+public double getCacheEvictionWatermark() {
+return cacheEvictionWatermark;
+}
+
public void clear() {
caches.values().forEach(EntryCache::clear);
}
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
index a66545fd3e5..c3e391ffab2 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
@@ -104,7 +104,7 @@ public class ManagedLedgerFactoryImpl implements
ManagedLedgerFactory {
private final ScheduledFuture statsTask;
private final
[pulsar-site] branch main updated: Fix more broken links
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 6c1161905d5 Fix more broken links
6c1161905d5 is described below
commit 6c1161905d57984bf7eee224f855d6c0816e62b1
Author: tison
AuthorDate: Thu Nov 17 10:36:51 2022 +0800
Fix more broken links
Signed-off-by: tison
---
site2/website-next/contribute/release-process.md | 4 ++--
site2/website-next/docusaurus.config.js | 8
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/site2/website-next/contribute/release-process.md
b/site2/website-next/contribute/release-process.md
index d3d2e4446ee..5483c2d105c 100644
--- a/site2/website-next/contribute/release-process.md
+++ b/site2/website-next/contribute/release-process.md
@@ -36,7 +36,7 @@ To verify the release branch is not broken, you can
synchronize the branch in yo
## Requirements
-If you haven't already done it, [create and publish the GPG
key](https://github.com/apache/pulsar/blob/master/wiki/release/create-gpg-keys.md)
to sign the release artifacts.
+If you haven't already done it, [create and publish the GPG
key](create-gpg-keys.md) to sign the release artifacts.
Before you start the next release steps, make sure you have installed the
**JDK8** and maven **3.6.1** for Pulsar 2.6 and Pulsar 2.7, and **JDK11** and
Maven **3.6.1** for Pulsar 2.8 onwards. And **clean up the bookkeeper's local
compiled** to make sure the bookkeeper dependency is fetched from the Maven
repo, details to see
https://lists.apache.org/thread/gsbh95b2d9xtcg5fmtxpm9k9q6w68gd2
@@ -438,7 +438,7 @@ Send out a PR request for review.
## Write release notes
-See [Pulsar Release Notes Guide](./release-note-guide.md).
+See [Pulsar Release Notes Guide](release-note-guide.md).
## Update the site
diff --git a/site2/website-next/docusaurus.config.js
b/site2/website-next/docusaurus.config.js
index d64914d8231..ad7ddf37eba 100644
--- a/site2/website-next/docusaurus.config.js
+++ b/site2/website-next/docusaurus.config.js
@@ -167,22 +167,22 @@ module.exports = {
label: "Pulsar Concepts",
},
{
+ to: `/docs/${versions[0]}/`,
label: "Quickstart",
- to: "/docs/",
},
{
- label: "Ecosystem",
to: "/ecosystem/",
+ label: "Ecosystem",
},
],
},
{
- href: `/docs/${versions[0]}/`,
+ to: `/docs/${versions[0]}/`,
position: "right",
label: "Docs",
},
{
- href: "/contribute/",
+ to: "/contribute/",
position: "right",
label: "Contribute",
},
[pulsar] annotated tag v2.11.0-candidate-1 updated (7ad0f5c8d68 -> 2ebc6dac8cb)
This is an automated email from the ASF dual-hosted git repository. technoboy pushed a change to annotated tag v2.11.0-candidate-1 in repository https://gitbox.apache.org/repos/asf/pulsar.git *** WARNING: tag v2.11.0-candidate-1 was modified! *** from 7ad0f5c8d68 (commit) to 2ebc6dac8cb (tag) tagging 7ad0f5c8d68f9fd5022895eca11c12a0a6d8b609 (commit) by Jiwe Guo on Thu Nov 17 10:12:34 2022 +0800 - Log - Release v2.11.0-candidate-1 -BEGIN PGP SIGNATURE- iQJJBAABCAAzFiEEStnMRiSmOggeXFwiZb2iVNVk0t8FAmN1mJQVHHRlY2hub2Jv eUBhcGFjaGUub3JnAAoJEGW9olTVZNLfFhgP/0ZoMd/qd6O697IBKEBXksAptTd6 7l3IDCQwfZeSSy/4eD/rKjAUQqXSJH0w8rgN6qi7Mxb3N7LDGAtlNaWzBfRnAWw6 aomtBs7kaKJfriC4JilmAJQjdZhl+84GJKhDjOlhsrE4QzRWBmW3fgKtd5Oht0Nh Yf3pqsUcaDmbKrqP9av2KdoWXNvUjwcjeA7zKW9NVu3BjsD0oQ9e0df0e+kZ/VnA 95nKTRYl/l5oENRrw3LyqHQiK38Y7PdzXsBHpb5/8pyHZ58JoUjnrRqsg0GUehNC p6bJAhVtqpxoOKTDLZbvia8El/ltEPuy15kiQkjMvHRYkElF40mZIiPSSJAk4ymv hpxZxve/jFLzQjTJ1PLvGakGiL7DSSA8pEl2kmHg0SU8MXgMMhWP7pKR7z5jlriH j1ydp1MkvOGcOX65gOWjZJZLumbFNFuAqofPC8HHkcseL0J3FNPZhx0vVL1MR+At v5ph1HkbdRMYwN/AuNbtdCg3Icz0XnV43dJWB9eOtCXrDqyTwyCuikY7mYy+0VGh SYM6nqwGafb1bqf2LZuD0VnZ5E27xxDKxIoilBtLnryEd4HBqc6Jy66+nFDB3pBp aJSpXj/I2taIvKVnc8xwyNe4/fiNhSVpTFxTNaN8nHlgUeExKooZzwORg5JFa0l4 WeM8AO7WvwB9TjXt =VPpl -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[GitHub] [pulsar] github-actions[bot] commented on pull request #17259: [wip][improve][broker] Supports assigning bundles to a given broker
github-actions[bot] commented on PR #17259: URL: https://github.com/apache/pulsar/pull/17259#issuecomment-1317958801 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #17375: [improve][admin] PulsarAdminBuilderImpl overrides timeout properties passed through config map
github-actions[bot] commented on PR #17375: URL: https://github.com/apache/pulsar/pull/17375#issuecomment-1317958711 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #17987: [Bug] Issue with implementing TLSAuthentication using Java
github-actions[bot] commented on issue #17987: URL: https://github.com/apache/pulsar/issues/17987#issuecomment-1317958266 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18502: [improve][broker] Unreasonable AuthenticationException reference
codecov-commenter commented on PR #18502: URL: https://github.com/apache/pulsar/pull/18502#issuecomment-1317958303 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18502?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#18502](https://codecov.io/gh/apache/pulsar/pull/18502?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (57b0574) into [master](https://codecov.io/gh/apache/pulsar/commit/fdf86d3b4e66e25936c2e2143bb5809b079ebc0d?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (fdf86d3) will **increase** coverage by `0.57%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/pulsar/pull/18502?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #18502 +/- ## + Coverage 45.67% 46.24% +0.57% + Complexity10075 9036-1039 Files 693 617 -76 Lines 6794058514-9426 Branches 7273 6099-1174 - Hits 3103027059-3971 + Misses328442-4891 + Partials 3577 3013 -564 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `46.24% <ø> (+0.57%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18502?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/pulsar/proxy/stats/TopicStats.java](https://codecov.io/gh/apache/pulsar/pull/18502/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc3RhdHMvVG9waWNTdGF0cy5qYXZh) | `58.82% <0.00%> (-41.18%)` | :arrow_down: | | [...g/apache/pulsar/broker/service/StreamingStats.java](https://codecov.io/gh/apache/pulsar/pull/18502/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1N0cmVhbWluZ1N0YXRzLmphdmE=) | `75.67% <0.00%> (-8.11%)` | :arrow_down: | | [...ava/org/apache/pulsar/utils/StatsOutputStream.java](https://codecov.io/gh/apache/pulsar/pull/18502/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL3V0aWxzL1N0YXRzT3V0cHV0U3RyZWFtLmphdmE=) | `65.45% <0.00%> (-5.46%)` | :arrow_down: | | [...oker/service/schema/SchemaRegistryServiceImpl.java](https://codecov.io/gh/apache/pulsar/pull/18502/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3NjaGVtYS9TY2hlbWFSZWdpc3RyeVNlcnZpY2VJbXBsLmphdmE=) | `57.95% <0.00%> (-5.11%)` | :arrow_down: | | [...er/impl/SingleSnapshotAbortedTxnProcessorImpl.java](https://codecov.io/gh/apache/pulsar/pull/18502/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvaW1wbC9TaW5nbGVTbmFwc2hvdEFib3J0ZWRUeG5Qcm9jZXNzb3JJbXBsLmphdmE=) | `51.19% <0.00%> (-3.58%)` | :arrow_down: | | [...sar/broker/service/schema/SchemaRegistryStats.java](https://codecov.io/gh/apache/pulsar/pull/18502/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3NjaGVtYS9TY2hlbWFSZWdpc3RyeVN0YXRzLmphdmE=) | `72.50% <0.00%> (-2.50%)` | :arrow_down: | |
[GitHub] [pulsar] github-actions[bot] commented on pull request #18007: [fix][broker] Reduce unnecessary persistence of markdelete
github-actions[bot] commented on PR #18007: URL: https://github.com/apache/pulsar/pull/18007#issuecomment-1317958218 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #18070: [fix][broker]correct doc of loadBalancerReportUpdateMaxIntervalMinutes
github-actions[bot] commented on PR #18070: URL: https://github.com/apache/pulsar/pull/18070#issuecomment-1317958144 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #18369: [improve][broker] System topic writer/reader connection not counted.
Technoboy- closed pull request #18369: [improve][broker] System topic writer/reader connection not counted. URL: https://github.com/apache/pulsar/pull/18369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Fix broken links (#297)
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new d580b0c36c9 Fix broken links (#297)
d580b0c36c9 is described below
commit d580b0c36c97ffd01c87df3d7cf52e1d2b4eea18
Author: tison
AuthorDate: Thu Nov 17 09:43:08 2022 +0800
Fix broken links (#297)
Signed-off-by: tison
---
site2/website-next/docusaurus.config.js | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/site2/website-next/docusaurus.config.js
b/site2/website-next/docusaurus.config.js
index 1d46132f5d2..d64914d8231 100644
--- a/site2/website-next/docusaurus.config.js
+++ b/site2/website-next/docusaurus.config.js
@@ -168,16 +168,16 @@ module.exports = {
},
{
label: "Quickstart",
- to: "/docs",
+ to: "/docs/",
},
{
label: "Ecosystem",
- to: "/ecosystem",
+ to: "/ecosystem/",
},
],
},
{
- href: `/docs/${versions[0]}`,
+ href: `/docs/${versions[0]}/`,
position: "right",
label: "Docs",
},
[GitHub] [pulsar-site] tisonkun merged pull request #297: Fix broken links
tisonkun merged PR #297: URL: https://github.com/apache/pulsar-site/pull/297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun opened a new pull request, #297: Fix broken links
tisonkun opened a new pull request, #297: URL: https://github.com/apache/pulsar-site/pull/297 Signed-off-by: tison -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][doc] Update links to the contribute guide (#18509)
This is an automated email from the ASF dual-hosted git repository. tison pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 6f6e32ec764 [improve][doc] Update links to the contribute guide (#18509) 6f6e32ec764 is described below commit 6f6e32ec764155b72cef08227de93c948283e939 Author: tison AuthorDate: Thu Nov 17 09:42:40 2022 +0800 [improve][doc] Update links to the contribute guide (#18509) Signed-off-by: tison Co-authored-by: momo-jun <60642177+momo-...@users.noreply.github.com> --- CONTRIBUTING.md | 4 +- README.md | 180 ++-- site2/README.md | 4 +- 3 files changed, 35 insertions(+), 153 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c57f9ab74b9..db83924c47c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,6 +21,4 @@ ## Contributing to Apache Pulsar -We would love for you to contribute to Apache Pulsar and make it even better! -Please check the [Contributing to Apache Pulsar](https://pulsar.apache.org/contributing/) -page before starting to work on the project. +We would love for you to contribute to Apache Pulsar and make it even better! Please check the [Apache Pulsar Contributing Guide](https://pulsar.apache.org/contribute/) before starting to work on the project. diff --git a/README.md b/README.md index f8220e1fbec..59911199f98 100644 --- a/README.md +++ b/README.md @@ -100,27 +100,27 @@ components in the Pulsar ecosystem, including connectors, adapters, and other la - pulsar ver > 2.10 and master branch -| Pulsar Components | Java Version | -| - | :---: | -| Broker| 17 | -| Functions / IO| 17 | -| CLI | 17 | -| Java Client | 8 or 11 or 17 | +| Components | Java Version | +||:-:| +| Broker | 17 | +| Functions / IO | 17 | +| CLI| 17 | +| Java Client| 8 or 11 or 17 | - 2.8 <= pulsar ver <= 2.10 -| Pulsar Components | Java Version | -| - | :--: | -| Broker| 11 | -| Functions / IO| 11 | -| CLI | 8 or 11| -| Java Client | 8 or 11| +| Components | Java Version | +||::| +| Broker | 11 | +| Functions / IO | 11 | +| CLI| 8 or 11| +| Java Client| 8 or 11| - pulsar ver < 2.8 -| Pulsar Components | Java Version | -| - | :--: | -| All | 8 or 11| +| Components | Java Version | +||::| +| All| 8 or 11| ## Build Pulsar @@ -128,11 +128,11 @@ components in the Pulsar ecosystem, including connectors, adapters, and other la - JDK - | Pulsar Version | JDK Version | - | - | :--: | - | master and 2.11 + | [JDK 17](https://adoptium.net/?variant=openjdk17) | - | 2.8 / 2.9 / 2.10 | [JDK 11](https://adoptium.net/?variant=openjdk11) | - | 2.7 - | [JDK 8](https://adoptium.net/?variant=openjdk8)| +| Pulsar Version|JDK Version| +|---|:-:| +| master and 2.11 + | [JDK 17](https://adoptium.net/?variant=openjdk17) | +| 2.8 / 2.9 / 2.10 | [JDK 11](https://adoptium.net/?variant=openjdk11) | +| 2.7 - | [JDK 8](https://adoptium.net/?variant=openjdk8) | - Maven 3.6.1+ - zip @@ -145,6 +145,7 @@ components in the Pulsar ecosystem, including connectors, adapters, and other la > It's better to use CMD rather than Powershell on Windows. Because maven will > activate the `windows` profile which runs `rename-netty-native-libs.cmd`. ### Build + Compile and install: ```bash @@ -159,7 +160,7 @@ $ mvn -pl module-name (e.g: pulsar-broker) install -DskipTests ### Minimal build (This skips most of external connectors and tiered storage handlers) -``` +```bash mvn install -Pcore-modules,-main -DskipTests ``` @@ -191,21 +192,18 @@ Check https://pulsar.apache.org for documentation and examples. ## Build custom docker images -Docker images must be built with Java 8 for `branch-2.7` or previous branches because of -[issue 8445](https://github.com/apache/pulsar/issues/8445). -Java 11 is the recommended JDK version in `branch-2.8`, `branch-2.9` and `branch-2.10`. -Java 17 is the recommended JDK version in `master`. +* Docker images must be built with Java 8 for `branch-2.7` or previous branches because of [ISSUE-8445](https://github.com/apache/pulsar/issues/8445). +* Java 11 is the recommended JDK version in `branch-2.8`, `branch-2.9` and `branch-2.10`. +* Java 17 is the recommended JDK version in `master`. -This builds the
[GitHub] [pulsar] tisonkun merged pull request #18509: [improve][doc] Update links to the contribute guide
tisonkun merged PR #18509: URL: https://github.com/apache/pulsar/pull/18509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on pull request #18509: [improve][doc] Update links to the contribute guide
tisonkun commented on PR #18509: URL: https://github.com/apache/pulsar/pull/18509#issuecomment-1317935436 Merging... Review after merge is welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Remove external configuration files (#296)
This is an automated email from the ASF dual-hosted git repository. tison pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 138de6e370e Remove external configuration files (#296) 138de6e370e is described below commit 138de6e370e8bf31a44d86cf2f035cfed8599c25 Author: tison AuthorDate: Thu Nov 17 09:19:26 2022 +0800 Remove external configuration files (#296) Signed-off-by: tison --- .../docs/reference-configuration-bookkeeper.md | 583 - .../docs/reference-configuration-log4j-shell.md| 37 -- .../docs/reference-configuration-log4j.md | 80 --- .../docs/reference-configuration-zookeeper.md | 83 --- 4 files changed, 783 deletions(-) diff --git a/site2/website-next/docs/reference-configuration-bookkeeper.md b/site2/website-next/docs/reference-configuration-bookkeeper.md deleted file mode 100644 index 28e85dcaf6c..000 --- a/site2/website-next/docs/reference-configuration-bookkeeper.md +++ /dev/null @@ -1,583 +0,0 @@ -# BookKeeper - -BookKeeper is a replicated log storage system that Pulsar uses for durable storage of all messages. - -### bookiePort - -The port on which the bookie server listens. - -**Default**: 3181 - -### allowLoopback - -Whether the bookie is allowed to use a loopback interface as its primary interface (that is the interface used to establish its identity). By default, loopback interfaces are not allowed to work as the primary interface. Using a loopback interface as the primary interface usually indicates a configuration error. For example, it’s fairly common in some VPS setups to not configure a hostname or to have the hostname resolve to `127.0.0.1`. If this is the case, then all bookies in the cluste [...] - -**Default**: false - -### listeningInterface - -The network interface on which the bookie listens. By default, the bookie listens on all interfaces. - -**Default**: eth0 - -### advertisedAddress - -Configure a specific hostname or IP address that the bookie should use to advertise itself to clients. By default, the bookie advertises either its own IP address or hostname according to the `listeningInterface` and `useHostNameAsBookieID` settings. - -**Default**: N/A - -### allowMultipleDirsUnderSameDiskPartition - -Configure the bookie to enable/disable multiple ledger/index/journal directories in the same filesystem disk partition. - -**Default**: false - -### minUsableSizeForIndexFileCreation - -The minimum safe usable size available in index directory for bookie to create index files while replaying journal at the time of bookie starts in Readonly Mode (in bytes). - -**Default**: 1073741824 - -### journalDirectory - -The directory where BookKeeper outputs its write-ahead log (WAL). - -**Default**: data/bookkeeper/journal - -### journalDirectories - -Directories that BookKeeper outputs its write ahead log. Multiple directories are available, being separated by `,`. For example: `journalDirectories=/tmp/bk-journal1,/tmp/bk-journal2`. If `journalDirectories` is set, the bookies skip `journalDirectory` and use this setting directory. - -**Default**: /tmp/bk-journal - -### ledgerDirectories - -The directory where BookKeeper outputs ledger snapshots. This could define multiple directories to store snapshots separated by `,`, for example `ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data`. Ideally, ledger dirs and the journal dir are each in a different device, which reduces the contention between random I/O and sequential write. It is possible to run with a single disk, but performance will be significantly lower. - -**Default**: data/bookkeeper/ledgers - -### ledgerManagerType - -The type of ledger manager used to manage how ledgers are stored, managed, and garbage collected. See [BookKeeper Internals](https://bookkeeper.apache.org/docs/next/getting-started/concepts) for more info. - -**Default**: hierarchical - -### zkLedgersRootPath - -The root ZooKeeper path used to store ledger metadata. This parameter is used by the ZooKeeper-based ledger manager as a root znode to store all ledgers. - -**Default**: /ledgers - -### ledgerStorageClass - -Ledger storage implementation class - -**Default**: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - -### entryLogFilePreallocationEnabled - -Enable or disable entry logger preallocation - -**Default**: true - -### logSizeLimit - -Max file size of the entry logger, in bytes. A new entry log file will be created when the old one reaches the file size limitation. - -**Default**: 1073741824 - -### minorCompactionThreshold - -Threshold of minor compaction. Entry log files whose remaining size percentage reaches below this threshold will be compacted in a minor compaction. If set to less than zero, the minor compaction is disabled. - -**Default**: 0.2 - -### minorCompactionInterval - -Time interval to run minor compaction,
[GitHub] [pulsar-site] tisonkun merged pull request #296: Remove external configuration files
tisonkun merged PR #296: URL: https://github.com/apache/pulsar-site/pull/296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun opened a new pull request, #296: Remove external configuration files
tisonkun opened a new pull request, #296: URL: https://github.com/apache/pulsar-site/pull/296 Trim contents according to https://github.com/apache/pulsar/pull/18508. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [cleanup][doc] Remove external configuration files (#18508)
This is an automated email from the ASF dual-hosted git repository. tison pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 8a08b5a7c22 [cleanup][doc] Remove external configuration files (#18508) 8a08b5a7c22 is described below commit 8a08b5a7c22bb849f07f9ead27482241f08f9e91 Author: tison AuthorDate: Thu Nov 17 09:16:44 2022 +0800 [cleanup][doc] Remove external configuration files (#18508) Signed-off-by: tison --- site2/docs/reference-configuration-bookkeeper.md | 583 -- site2/docs/reference-configuration-log4j-shell.md | 37 -- site2/docs/reference-configuration-log4j.md | 80 --- site2/docs/reference-configuration-zookeeper.md | 83 --- 4 files changed, 783 deletions(-) diff --git a/site2/docs/reference-configuration-bookkeeper.md b/site2/docs/reference-configuration-bookkeeper.md deleted file mode 100644 index 28e85dcaf6c..000 --- a/site2/docs/reference-configuration-bookkeeper.md +++ /dev/null @@ -1,583 +0,0 @@ -# BookKeeper - -BookKeeper is a replicated log storage system that Pulsar uses for durable storage of all messages. - -### bookiePort - -The port on which the bookie server listens. - -**Default**: 3181 - -### allowLoopback - -Whether the bookie is allowed to use a loopback interface as its primary interface (that is the interface used to establish its identity). By default, loopback interfaces are not allowed to work as the primary interface. Using a loopback interface as the primary interface usually indicates a configuration error. For example, it’s fairly common in some VPS setups to not configure a hostname or to have the hostname resolve to `127.0.0.1`. If this is the case, then all bookies in the cluste [...] - -**Default**: false - -### listeningInterface - -The network interface on which the bookie listens. By default, the bookie listens on all interfaces. - -**Default**: eth0 - -### advertisedAddress - -Configure a specific hostname or IP address that the bookie should use to advertise itself to clients. By default, the bookie advertises either its own IP address or hostname according to the `listeningInterface` and `useHostNameAsBookieID` settings. - -**Default**: N/A - -### allowMultipleDirsUnderSameDiskPartition - -Configure the bookie to enable/disable multiple ledger/index/journal directories in the same filesystem disk partition. - -**Default**: false - -### minUsableSizeForIndexFileCreation - -The minimum safe usable size available in index directory for bookie to create index files while replaying journal at the time of bookie starts in Readonly Mode (in bytes). - -**Default**: 1073741824 - -### journalDirectory - -The directory where BookKeeper outputs its write-ahead log (WAL). - -**Default**: data/bookkeeper/journal - -### journalDirectories - -Directories that BookKeeper outputs its write ahead log. Multiple directories are available, being separated by `,`. For example: `journalDirectories=/tmp/bk-journal1,/tmp/bk-journal2`. If `journalDirectories` is set, the bookies skip `journalDirectory` and use this setting directory. - -**Default**: /tmp/bk-journal - -### ledgerDirectories - -The directory where BookKeeper outputs ledger snapshots. This could define multiple directories to store snapshots separated by `,`, for example `ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data`. Ideally, ledger dirs and the journal dir are each in a different device, which reduces the contention between random I/O and sequential write. It is possible to run with a single disk, but performance will be significantly lower. - -**Default**: data/bookkeeper/ledgers - -### ledgerManagerType - -The type of ledger manager used to manage how ledgers are stored, managed, and garbage collected. See [BookKeeper Internals](https://bookkeeper.apache.org/docs/next/getting-started/concepts) for more info. - -**Default**: hierarchical - -### zkLedgersRootPath - -The root ZooKeeper path used to store ledger metadata. This parameter is used by the ZooKeeper-based ledger manager as a root znode to store all ledgers. - -**Default**: /ledgers - -### ledgerStorageClass - -Ledger storage implementation class - -**Default**: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - -### entryLogFilePreallocationEnabled - -Enable or disable entry logger preallocation - -**Default**: true - -### logSizeLimit - -Max file size of the entry logger, in bytes. A new entry log file will be created when the old one reaches the file size limitation. - -**Default**: 1073741824 - -### minorCompactionThreshold - -Threshold of minor compaction. Entry log files whose remaining size percentage reaches below this threshold will be compacted in a minor compaction. If set to less than zero, the minor compaction is disabled. - -**Default**: 0.2 - -### minorCompactionInterval - -Time interval to run minor compaction, in
[GitHub] [pulsar] tisonkun merged pull request #18508: [cleanup][doc] Remove external configuration files
tisonkun merged PR #18508: URL: https://github.com/apache/pulsar/pull/18508 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on pull request #18509: [improve][doc] Update links to the contribute guide
tisonkun commented on PR #18509: URL: https://github.com/apache/pulsar/pull/18509#issuecomment-1317915451 @momo-jun Thanks! Comments addressed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18509: [improve][doc] Update links to the contribute guide
momo-jun commented on code in PR #18509: URL: https://github.com/apache/pulsar/pull/18509#discussion_r1024656007 ## README.md: ## @@ -191,21 +192,18 @@ Check https://pulsar.apache.org for documentation and examples. ## Build custom docker images -Docker images must be built with Java 8 for `branch-2.7` or previous branches because of -[issue 8445](https://github.com/apache/pulsar/issues/8445). -Java 11 is the recommended JDK version in `branch-2.8`, `branch-2.9` and `branch-2.10`. -Java 17 is the recommended JDK version in `master`. +* Docker images must be built with Java 8 for `branch-2.7` or previous branches because of [ISSUE-8445](https://github.com/apache/pulsar/issues/8445). +* Java 11 is the recommended JDK version in `branch-2.8`, `branch-2.9` and `branch-2.10`. +* Java 17 is the recommended JDK version in `master`. -This builds the docker images `apachepulsar/pulsar-all:latest` and `apachepulsar/pulsar:latest`. +The following command builds the docker images `apachepulsar/pulsar-all:latest` and `apachepulsar/pulsar:latest`: ```bash mvn clean install -DskipTests mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests ``` -After the images are built, they can be tagged and pushed to your custom repository. -Here's an example of a bash script that tags the docker images with the current version and git revision and -pushes them to `localhost:32000/apachepulsar`. +After the images are built, they can be tagged and pushed to your custom repository. Here's an example of a bash script that tags the docker images with the current version and git revision and pushes them to `localhost:32000/apachepulsar`. Review Comment: ```suggestion After the images are built, they can be tagged and pushed to your custom repository. Here's an example of a bash script that tags the docker images with the current version and git revision and pushes them to `localhost:32000/apachepulsar`. ``` ## CONTRIBUTING.md: ## @@ -21,6 +21,4 @@ ## Contributing to Apache Pulsar -We would love for you to contribute to Apache Pulsar and make it even better! -Please check the [Contributing to Apache Pulsar](https://pulsar.apache.org/contributing/) -page before starting to work on the project. +We would love for you to contribute to Apache Pulsar and make it even better! Please check the [Apache Pulsar Contributing Guide](https://pulsar.apache.org/contribute/) page before starting to work on the project. Review Comment: ```suggestion We would love for you to contribute to Apache Pulsar and make it even better! Please check the [Apache Pulsar Contributing Guide](https://pulsar.apache.org/contribute/) before starting to work on the project. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun closed issue #17442: [Bug] Zookeeper TLS breaks /bin/pulsar zookeeper-shell
tisonkun closed issue #17442: [Bug] Zookeeper TLS breaks /bin/pulsar zookeeper-shell URL: https://github.com/apache/pulsar/issues/17442 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #17442: [Bug] Zookeeper TLS breaks /bin/pulsar zookeeper-shell
tisonkun commented on issue #17442: URL: https://github.com/apache/pulsar/issues/17442#issuecomment-1317904534 Closed as no response. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #17657: Tracking issue for slow tests
tisonkun commented on issue #17657: URL: https://github.com/apache/pulsar/issues/17657#issuecomment-1317903520 All subtasks are fixed. Closed as done. If you find more slow tests after a recheck, feel free to reopen the issue. Or prefer to open a new issue to another turn. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun closed issue #17657: Tracking issue for slow tests
tisonkun closed issue #17657: Tracking issue for slow tests URL: https://github.com/apache/pulsar/issues/17657 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Update Python API document generate guide ref
This is an automated email from the ASF dual-hosted git repository. tison pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 3a0e25997a2 Update Python API document generate guide ref 3a0e25997a2 is described below commit 3a0e25997a27b853ce3073ff6593183c3577275f Author: tison AuthorDate: Thu Nov 17 08:56:50 2022 +0800 Update Python API document generate guide ref Signed-off-by: tison --- site2/website-next/contribute/release-process.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/website-next/contribute/release-process.md b/site2/website-next/contribute/release-process.md index 6de8ec445dc..d3d2e4446ee 100644 --- a/site2/website-next/contribute/release-process.md +++ b/site2/website-next/contribute/release-process.md @@ -408,7 +408,7 @@ Then you can run `twin upload` to upload those wheel files. After publishing the python client docs, run the following script from the apache/pulsar-site `main` branch: ```shell -PULSAR_VERSION=2.X.Y ./site2/tools/api/python/build-docs-in-docker.sh +PULSAR_VERSION=2.X.Y ./site2/tools/apidoc/python/pdoc-generator.sh ``` Note that it builds the docs within a docker image, so you'll need to have docker running.
[GitHub] [pulsar] tisonkun commented on a diff in pull request #18504: [improve][ci] Allow [fix][broker][branch-2.8] alike patten
tisonkun commented on code in PR #18504: URL: https://github.com/apache/pulsar/pull/18504#discussion_r1024642578 ## .github/workflows/ci-semantic-pull-request.yml: ## @@ -61,8 +61,8 @@ jobs: # bk -> bookkeeper scopes: | admin +bk Review Comment: @michaeljmarshall I find that you add `bk` and `zk` scopes. There're several overlapping scopes now: * `zk` vs `meta` * `bk` vs `ml` vs `storage` How do you understand this situation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #18504: [improve][ci] Allow [fix][broker][branch-2.8] alike patten
tisonkun commented on code in PR #18504: URL: https://github.com/apache/pulsar/pull/18504#discussion_r1024642578 ## .github/workflows/ci-semantic-pull-request.yml: ## @@ -61,8 +61,8 @@ jobs: # bk -> bookkeeper scopes: | admin +bk Review Comment: @michaeljmarshall I find that you add `bk` and `zk` scopes. There're several overlapping scopes now: * `zk` vs `meta` * `bk` vs `ml` vs `storage` How can you understand this situation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun merged pull request #18504: [improve][ci] Allow [fix][broker][branch-2.8] alike patten
tisonkun merged PR #18504: URL: https://github.com/apache/pulsar/pull/18504 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][ci] Allow [fix][broker][branch-2.8] alike patten (#18504)
This is an automated email from the ASF dual-hosted git repository. tison pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 55d7debae47 [improve][ci] Allow [fix][broker][branch-2.8] alike patten (#18504) 55d7debae47 is described below commit 55d7debae47d1230a70968e3ba48ce1bcfacb0db Author: tison AuthorDate: Thu Nov 17 08:35:01 2022 +0800 [improve][ci] Allow [fix][broker][branch-2.8] alike patten (#18504) Signed-off-by: tison --- .github/workflows/ci-semantic-pull-request.yml | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci-semantic-pull-request.yml b/.github/workflows/ci-semantic-pull-request.yml index bbfa7ba3668..8fb45c54b02 100644 --- a/.github/workflows/ci-semantic-pull-request.yml +++ b/.github/workflows/ci-semantic-pull-request.yml @@ -61,8 +61,8 @@ jobs: # bk -> bookkeeper scopes: | admin +bk broker -ml build ci cli @@ -72,6 +72,7 @@ jobs: io meta misc +ml monitor offload proxy @@ -84,13 +85,12 @@ jobs: txn ws zk -bk - # The pull request's title should be fulfill the following pattern: + # The pull request's title should be fulfilled the following pattern: # # [][] # # ... where valid types and scopes can be found above; for example: # # [fix][test] flaky test V1_ProxyAuthenticationTest.anonymousSocketTest - headerPattern: '^\[(\w*)\](?:\[(.*)\])? (.*)$' + headerPattern: '^\[(\w*?)\](?:\[(.*?)\])?(:?\s*)(.*)$' headerPatternCorrespondence: type, scope, subject
[GitHub] [pulsar] tisonkun commented on pull request #18504: [improve][ci] Allow [fix][broker][branch-2.8] alike patten
tisonkun commented on PR #18504: URL: https://github.com/apache/pulsar/pull/18504#issuecomment-1317876758 Merging... Verified on https://regex101.com/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch asf-site-next updated: Manually Trim
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch asf-site-next
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/asf-site-next by this push:
new af84ca5a1cb Manually Trim
af84ca5a1cb is described below
commit af84ca5a1cb97753413a075904e8772fe1cda378
Author: tison
AuthorDate: Thu Nov 17 08:17:20 2022 +0800
Manually Trim
Signed-off-by: tison
---
.../contribute/category/committer-guide/index.html | 38 -
content/contribute/category/releasing/index.html | 38 -
.../become-a-committer-or-pmc-member/index.html| 38 -
.../documentation/contribution/index.html | 38 -
content/contribute/documentation/label/index.html | 38 -
content/contribute/documentation/naming/index.html | 38 -
.../contribute/documentation/overview/index.html | 38 -
.../contribute/documentation/preview/index.html| 38 -
content/contribute/documentation/syntax/index.html | 38 -
.../getting-started/setup-building/index.html | 38 -
.../getting-started/setup-ide/index.html | 38 -
.../releasing/create-gpg-keys/index.html | 38 -
.../releasing/release-note-guide/index.html| 38 -
.../releasing/release-process/index.html | 66 --
.../validate-release-candidate/index.html | 38 -
.../testing-and-ci/ci-testing-in-fork/index.html | 38 -
.../contribute/testing-and-ci/licensing/index.html | 38 -
17 files changed, 674 deletions(-)
diff --git a/content/contribute/category/committer-guide/index.html
b/content/contribute/category/committer-guide/index.html
deleted file mode 100644
index c89441cc443..000
--- a/content/contribute/category/committer-guide/index.html
+++ /dev/null
@@ -1,38 +0,0 @@
-
-
-
-
-
-Committer Guide | Apache Pulsarhttps://pulsar.apache.org/contribute/category/committer-guide;>
-
-https://www.google-analytics.com;>
-window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)},ga.l=+new
Date,ga("create","UA-102219959-1","auto"),ga("send","pageview")
-https://www.google-analytics.com/analytics.js";>
-
-
-
-
-
-
-
-
-https://fonts.googleapis.com/css2?family=Red+Hat+Display:wght@300;400;600,900=swap;>
-
-
-
-
-
-
-!function(){function
t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var
t=null;try{t=localStorage.getItem("theme")}catch(t){}return
t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())
-Skip to
main content Pulsar Summit Asia 2022 will take place on
November 19th and 20th, 2022. https://pulsar.apache.org/blog/2022/11/0 [...]
-
- http://www.apache.org/; target="_blank" rel="noopener noreferrer"
class="footer__link-item">Foundationhttps://www.apache.org/events/current- [...]
- Apache Pulsar is available under the
https://www.apache.org/licenses/LICENSE-2.0; target="_blank">Apache
License, version 2.0.
-Apache Pulsar is a distributed, open source pub-sub
messaging and streaming platform for real-time workloads, managing hundreds of
billions of events per day.
- Apache Pulsar is available under the Apache
License, version 2.0.
- Copyright © 2022 The Apache Software Foundation. All Rights Reserved.
Apache, Pulsar, Apache Pulsar, and the Apache feather logo are trademarks or
registered trademarks of The Apache Software
Foundation.
-
-
-
-
\ No newline at end of file
diff --git a/content/contribute/category/releasing/index.html
b/content/contribute/category/releasing/index.html
deleted file mode 100644
index 1d1093b5e74..000
--- a/content/contribute/category/releasing/index.html
+++ /dev/null
@@ -1,38 +0,0 @@
-
-
-
-
-
-Releasing | Apache Pulsarhttps://pulsar.apache.org/contribute/category/releasing;>
-
-https://www.google-analytics.com;>
-window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)},ga.l=+new
Date,ga("create","UA-102219959-1","auto"),ga("send","pageview")
-https://www.google-analytics.com/analytics.js";>
-
-
-
-
-
-
-
-
-https://fonts.googleapis.com/css2?family=Red+Hat+Display:wght@300;400;600,900=swap;>
-
-
-
-
-
-
-!function(){function
t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var
t=null;try{t=localStorage.getItem("theme")}catch(t){}return
t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())
-Skip to
main content Pulsar Summit Asia 2022
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#76308f0)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 49235ccc27e Docs sync done from apache/pulsar(#76308f0) 49235ccc27e is described below commit 49235ccc27e9f0a910ddee1d02e9a85b8b2c57a3 Author: Pulsar Site Updater AuthorDate: Thu Nov 17 00:02:08 2022 + Docs sync done from apache/pulsar(#76308f0)
[GitHub] [pulsar] heesung-sn commented on pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
heesung-sn commented on PR #18489: URL: https://github.com/apache/pulsar/pull/18489#issuecomment-1317844321 this PR implements https://github.com/apache/pulsar/pull/18079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #18494: Flatten files (slug) and use a siderbar config to categorize documents
tisonkun commented on issue #18494: URL: https://github.com/apache/pulsar/issues/18494#issuecomment-1317838995 Reverse links: * https://github.com/apache/pulsar/pull/18508 * https://github.com/apache/pulsar/pull/18509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun closed issue #18494: Flatten files (slug) and use a siderbar config to categorize documents
tisonkun closed issue #18494: Flatten files (slug) and use a siderbar config to categorize documents URL: https://github.com/apache/pulsar/issues/18494 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #18494: Flatten files (slug) and use a siderbar config to categorize documents
tisonkun commented on issue #18494: URL: https://github.com/apache/pulsar/issues/18494#issuecomment-1317836714 Closed by https://github.com/apache/pulsar-site/pull/295. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #18509: [improve][doc] Update links to the contribute guide
tisonkun commented on code in PR #18509:
URL: https://github.com/apache/pulsar/pull/18509#discussion_r1024619869
##
README.md:
##
@@ -221,134 +219,20 @@ docker push ${image_repo_and_project}/pulsar:$tag
## Setting up your IDE
Review Comment:
I agree that it helps if we inline this info but it's too long to keep in
sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] tisonkun opened a new pull request, #18509: [improve][doc] Update links to the contribute guide
tisonkun opened a new pull request, #18509: URL: https://github.com/apache/pulsar/pull/18509 ### Motivation This reflects https://github.com/apache/pulsar-site/pull/295. ### Documentation - [x] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #18177: [improve][build] use SLASHSTAR_STYLE for licenses in Java files
mattisonchao commented on PR #18177: URL: https://github.com/apache/pulsar/pull/18177#issuecomment-1317830296 Ok, Got it. thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] arpillai89 commented on issue #13167: Not enough non-faulty bookies available
arpillai89 commented on issue #13167: URL: https://github.com/apache/pulsar/issues/13167#issuecomment-1317827145 @moweonlee , could you please let me know what exact config/value did you update for retention and TTL to get this issue resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun opened a new pull request, #18508: [cleanup][doc] Remove external configuration files
tisonkun opened a new pull request, #18508: URL: https://github.com/apache/pulsar/pull/18508 ### Motivation This is the counterpart of https://github.com/apache/pulsar-site/pull/295/commits/2cafa154aad083d5646dfb1bc40ec8120e15e1a4 which acts as a follow-up to https://github.com/apache/pulsar-site/pull/259. ### Documentation - [x] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun merged pull request #295: Refactor contribution guide
tisonkun merged PR #295: URL: https://github.com/apache/pulsar-site/pull/295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18504: [improve][ci] Allow [fix][broker][branch-2.8] alike patten
codecov-commenter commented on PR #18504: URL: https://github.com/apache/pulsar/pull/18504#issuecomment-1317799146 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18504?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#18504](https://codecov.io/gh/apache/pulsar/pull/18504?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0f1bb2c) into [master](https://codecov.io/gh/apache/pulsar/commit/79750231051db849350e3d35dd3706320466acac?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (7975023) will **decrease** coverage by `4.05%`. > The diff coverage is `59.37%`. [](https://codecov.io/gh/apache/pulsar/pull/18504?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #18504 +/- ## - Coverage 45.62% 41.56% -4.06% + Complexity10075 2404-7671 Files 697 234 -463 Lines 6802416672 -51352 Branches 7293 1827-5466 - Hits 31033 6929 -24104 + Misses33413 8988 -24425 + Partials 3578 755-2823 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `41.56% <59.37%> (-4.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18504?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...nt/impl/PulsarClientImplementationBindingImpl.java](https://codecov.io/gh/apache/pulsar/pull/18504/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1B1bHNhckNsaWVudEltcGxlbWVudGF0aW9uQmluZGluZ0ltcGwuamF2YQ==) | `72.41% <ø> (-0.47%)` | :arrow_down: | | [...ar/client/impl/conf/ConsumerConfigurationData.java](https://codecov.io/gh/apache/pulsar/pull/18504/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL2NvbmYvQ29uc3VtZXJDb25maWd1cmF0aW9uRGF0YS5qYXZh) | `92.55% <ø> (+2.12%)` | :arrow_up: | | [...he/pulsar/client/impl/TypedMessageBuilderImpl.java](https://codecov.io/gh/apache/pulsar/pull/18504/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1R5cGVkTWVzc2FnZUJ1aWxkZXJJbXBsLmphdmE=) | `30.76% <59.37%> (+3.56%)` | :arrow_up: | | [...apache/pulsar/proxy/server/LookupProxyHandler.java](https://codecov.io/gh/apache/pulsar/pull/18504/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc2VydmVyL0xvb2t1cFByb3h5SGFuZGxlci5qYXZh) | `57.75% <0.00%> (-0.44%)` | :arrow_down: | | [...n/coordinator/TransactionCoordinatorException.java](https://codecov.io/gh/apache/pulsar/pull/18504/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9leGNlcHRpb24vY29vcmRpbmF0b3IvVHJhbnNhY3Rpb25Db29yZGluYXRvckV4Y2VwdGlvbi5qYXZh) | | | | [...ar/broker/loadbalance/impl/SimpleResourceUnit.java](https://codecov.io/gh/apache/pulsar/pull/18504/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9sb2FkYmFsYW5jZS9pbXBsL1NpbXBsZVJlc291cmNlVW5pdC5qYXZh) | | | |
[GitHub] [pulsar] aymkhalil opened a new issue, #18507: [Bug] Pulsar Admin does not update "inputSpecs" on sinks
aymkhalil opened a new issue, #18507:
URL: https://github.com/apache/pulsar/issues/18507
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues)
and found nothing similar.
### Version
2.10 on M1 mac
### Minimal reproduce step
* Create any sink without input specs:
```
bin/pulsar-admin sinks create --name print-sink --archive
pulsar-io/data-generator/target/pulsar-io-data-generator-2.10.2.3-SNAPSHOT.nar
--inputs persistent://public/default/fresh
```
* Try to update the sink with `inputSpecs`
```
bin/pulsar-admin sinks update --name print-sink --archive
pulsar-io/data-generator/target/pulsar-io-data-generator-2.10.2.3-SNAPSHOT.nar
--inputs persistent://public/default/fresh --input-specs
'{"persistent://public/default/fresh":{"receiverQueueSize":10}}'
```
* Get the effective sink config:
```
bin/pulsar-admin sinks get --name print-sink
```
### What did you expect to see?
```
{
"tenant": "public",
"namespace": "default",
"name": "print-sink",
"className": "org.apache.pulsar.io.datagenerator.DataGeneratorPrintSink",
"sourceSubscriptionPosition": "Latest",
"inputs": [
"persistent://public/default/fresh"
],
"inputSpecs": {
"persistent://public/default/fresh": {
"isRegexPattern": false,
"schemaProperties": {},
"consumerProperties": {},
"receiverQueueSize": 10,
"poolMessages": false
}
},
...
}
```
### What did you see instead?
`receiverQueueSize` is missing from `inputSpecs`
```
{
"tenant": "public",
"namespace": "default",
"name": "print-sink",
"className": "org.apache.pulsar.io.datagenerator.DataGeneratorPrintSink",
"sourceSubscriptionPosition": "Latest",
"inputs": [
"persistent://public/default/fresh"
],
"inputSpecs": {
"persistent://public/default/fresh": {
"isRegexPattern": false,
"schemaProperties": {},
"consumerProperties": {},
"poolMessages": false
}
},
...
}
```
### Anything else?
Please note that the create sink command with `inputSpecs` works fine - the
issue is with `update` only:
```
bin/pulsar-admin sinks create --name print-sink --archive
pulsar-io/data-generator/target/pulsar-io-data-generator-2.10.2.3-SNAPSHOT.nar
--inputs persistent://public/default/fresh --input-specs
'{"persistent://public/default/fresh":{"receiverQueueSize":10}}'
bin/pulsar-admin sinks get --name print-sink
"inputSpecs": {
"persistent://public/default/fresh": {
"isRegexPattern": false,
"schemaProperties": {},
"consumerProperties": {},
"receiverQueueSize": 10,
"poolMessages": false
}
},
```
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #18504: [improve][ci] Allow [fix][broker][branch-2.9] alike patten
tisonkun commented on code in PR #18504: URL: https://github.com/apache/pulsar/pull/18504#discussion_r1024581332 ## .github/workflows/ci-semantic-pull-request.yml: ## @@ -84,13 +85,12 @@ jobs: txn ws zk -bk - # The pull request's title should be fulfill the following pattern: + # The pull request's title should be fulfilled the following pattern: # # [][] # # ... where valid types and scopes can be found above; for example: # # [fix][test] flaky test V1_ProxyAuthenticationTest.anonymousSocketTest - headerPattern: '^\[(\w*)\](?:\[(.*)\])? (.*)$' + headerPattern: '^\[(\w*)\](?:\[(.*)\])?(:?\s*)(.*)$' Review Comment: ```suggestion headerPattern: '^\[(\w*?)\](?:\[(.*?)\])?(:?\s*)(.*)$' ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on a diff in pull request #18451: [fix][broker] Correctly set byte and message out totals per subscription
michaeljmarshall commented on code in PR #18451:
URL: https://github.com/apache/pulsar/pull/18451#discussion_r1024425379
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##
@@ -65,11 +65,19 @@ void releaseAll() {
metricStreamMap.clear();
}
-private SimpleTextOutputStream initGaugeType(String metricName) {
+private SimpleTextOutputStream initMetricType(String metricName) {
+String metricTypeDef = String.format("# TYPE %s %s\n", metricName,
metricType(metricName));
Review Comment:
Nit: this might have unintended consequences for performance. With this
change, we're creating the string for every call to this method instead of only
when the metric has not yet been initialized. I'm not sure how much it matters
in this section, but I know the prometheus metrics generation has been
optimized because it was generating unnecessary load.
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##
@@ -65,11 +65,19 @@ void releaseAll() {
metricStreamMap.clear();
}
-private SimpleTextOutputStream initGaugeType(String metricName) {
+private SimpleTextOutputStream initMetricType(String metricName) {
+String metricTypeDef = String.format("# TYPE %s %s\n", metricName,
metricType(metricName));
return metricStreamMap.computeIfAbsent(metricName, s -> {
SimpleTextOutputStream stream = new
SimpleTextOutputStream(PulsarByteBufAllocator.DEFAULT.directBuffer());
-stream.write("# TYPE ").write(metricName).write(" gauge\n");
+stream.write(metricTypeDef);
return stream;
});
}
+
+private String metricType(String metricName) {
+if (metricName.endsWith("_total")) {
+return "counter";
+}
+return "gauge";
+}
Review Comment:
I wonder if this change should be a separate, larger PR. I agree that we
need to fix the `TYPE` annotations so that things are not all marked as
`gauge`, but it seems like a separate issue than fixing the bytes metric. (The
main reason I think we should split it out is so we can create an optimized
solution. I am wondering if it would be better to create an enum, or use
Prometheus's enums, and have the caller indicate the metric type.) Also, is
there any way changing this annotation will "break" user integration? If so,
we'll want to make sure we correctly notify users that this bug fix will change
the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] heesung-sn opened a new issue, #18506: [Doc] Improve Pulsar Pub/Sub Throttling Doc
heesung-sn opened a new issue, #18506: URL: https://github.com/apache/pulsar/issues/18506 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### What issue do you find in Pulsar docs? The current documentation about Pulsar message pub/sub throttling is insufficient. ### What is your suggestion? - Have a central page about Pulsar message throttling. - Explain the goal of Pulsar message throttling. - Explain how and where Pulsar throttling works. - Explain what configs and admin APIs are available to set throttling. - Explain how throttling broker configs and admin APIs are working together if they are configured at the same time. (explain config-adminAPI compatibility) - Explain How throttling works with different features: message batching, compression, chunking, backlog, replicas, entry filter and etc. (explain feature compatibility) - Give example setups for best throttling practices ### Any reference? https://pulsar.apache.org/docs/2.10.x/admin-api-namespaces/#configure-dispatch-throttling-for-topics pulsar % grep Throttling conf/broker.conf topicPublisherThrottlingTickTimeMillis=10 brokerPublisherThrottlingTickTimeMillis=50 brokerPublisherThrottlingMaxMessageRate=0 brokerPublisherThrottlingMaxByteRate=0 subscribeThrottlingRatePerConsumer=0 dispatchThrottlingRateInMsg=0 dispatchThrottlingRateInByte=0 dispatchThrottlingRatePerTopicInMsg=0 dispatchThrottlingRatePerTopicInByte=0 dispatchThrottlingOnBatchMessageEnabled=false dispatchThrottlingRatePerSubscriptionInMsg=0 dispatchThrottlingRatePerSubscriptionInByte=0 dispatchThrottlingRatePerReplicatorInMsg=0 dispatchThrottlingRatePerReplicatorInByte=0 dispatchThrottlingRateRelativeToPublishRate=false dispatchThrottlingOnNonBacklogConsumerEnabled=true dispatchThrottlingForFilteredEntriesEnabled=false Some references regarding the above configs: https://github.com/apache/pulsar/pull/5710 https://github.com/apache/pulsar/pull/11325 https://github.com/apache/pulsar/pull/2977 https://github.com/apache/pulsar/pull/754 https://github.com/apache/pulsar/pull/4273 https://github.com/apache/pulsar/pull/5804 https://github.com/apache/pulsar/pull/12294 https://github.com/apache/pulsar/pull/17686 ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #18505: [refactor][broker] Optimize semantics
lhotari commented on PR #18505: URL: https://github.com/apache/pulsar/pull/18505#issuecomment-1317515571 @crossoverJie Please follow the instructions in https://github.com/apache/pulsar/actions/runs/3481081589/attempts/1#summary-9529786678 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] rdhabalia commented on a diff in pull request #10601: Fix getting partition metadata of a nonexistent topic returns 0
rdhabalia commented on code in PR #10601:
URL: https://github.com/apache/pulsar/pull/10601#discussion_r1024344326
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java:
##
@@ -543,6 +543,21 @@ protected PartitionedTopicMetadata
internalGetPartitionedMetadata(boolean author
boolean
checkAllowAutoCreation) {
PartitionedTopicMetadata metadata =
getPartitionedTopicMetadata(topicName,
authoritative, checkAllowAutoCreation);
+if (metadata.partitions == 0 && !checkAllowAutoCreation) {
+// The topic may be a non-partitioned topic, so check if it exists
here.
+// However, when checkAllowAutoCreation is true, the client will
create the topic if it doesn't exist.
+// In this case, `partitions == 0` means the automatically created
topic is a non-partitioned topic so we
+// shouldn't check if the topic exists.
+try {
+if
(!pulsar().getNamespaceService().checkTopicExists(topicName).get()) {
+throw new RestException(Status.NOT_FOUND,
Review Comment:
this breaks HTTP lookup for all the previous clients and also breaks the
semantic of topic creation. why are we creating new semantics by breaking the
compatibility? what are we trying to achieve here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] rdhabalia commented on pull request #10601: Fix getting partition metadata of a nonexistent topic returns 0
rdhabalia commented on PR #10601: URL: https://github.com/apache/pulsar/pull/10601#issuecomment-1317441807 This PR breaks all previous clients who would like to use HTTP lookup. HTTP lookup at client side expects partitioned metadata with 0 partition and this PR gives `TopicNotFoundException` to client and old client couldn't handle it and won't be able to create a new topic which breaks topic creation semantics of Pulsar. I have seen multiple such instances in past where we broke the compatibility and it's not acceptable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18491: [fix][client] Fix multi-topic consumer stuck after redeliver messages
poorbarcode commented on code in PR #18491:
URL: https://github.com/apache/pulsar/pull/18491#discussion_r1024337980
##
pulsar-broker/src/test/java/org/apache/pulsar/client/impl/NegativeAcksTest.java:
##
@@ -439,4 +443,61 @@ public void run() {
Assert.assertEquals(count, 9);
Assert.assertEquals(0, datas.size());
}
+
+/**
+ * see https://github.com/apache/pulsar/pull/18491
+ */
+@Test
+public void testMultiTopicConsumerConcurrentRedeliverAndReceive() throws
Exception {
Review Comment:
Already add a sleep before receiving messages.
I have tried to increase the probability of test failure, but can't get to
100%. Because that:
- method `receiveMessageFromConsumer` will call
`consumerImpl.batchReceiveAsync` and `consumerImpl.receiveAsync` .
- When `consumerImpl.batchReceiveAsync` and `consumerImpl.receiveAsync` is
finished, it will trigger another `receiveMessageFromConsumer`.
- `consumerImpl.batchReceiveAsync` and `consumerImpl.receiveAsync` calls
each other
- if receive messages is empty call `consumerImpl.receiveAsync`
- else call `consumerImpl.batchReceiveAsync`
- `consumerImpl.batchReceiveAsync` and `consumerImpl.receiveAsync` executed
asynchronously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#76308f0)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new a12506b4c4d Docs sync done from apache/pulsar(#76308f0) a12506b4c4d is described below commit a12506b4c4d505dfda84cf04ca17066a46a05f18 Author: Pulsar Site Updater AuthorDate: Wed Nov 16 18:01:47 2022 + Docs sync done from apache/pulsar(#76308f0)
[GitHub] [pulsar-helm-chart] briedel opened a new issue, #335: Proxy fails to connect to port 80
briedel opened a new issue, #335:
URL: https://github.com/apache/pulsar-helm-chart/issues/335
When using the helm chart on GKE, the proxy is unable to connect to port 80,
which crashes the proxy pod and begins a restart loop. The logs how this
```
Defaulted container "pulsar-300-proxy" out of: pulsar-300-proxy,
wait-zookeeper-ready (init), wait-broker-ready (init)
[conf/proxy.conf] Applying config authenticationEnabled = true
[conf/proxy.conf] Applying config authenticationProviders =
org.apache.pulsar.broker.authentication.AuthenticationProviderToken
[conf/proxy.conf] Applying config authorizationEnabled = false
[conf/proxy.conf] Applying config brokerClientAuthenticationParameters =
file:///pulsar/tokens/proxy/token
[conf/proxy.conf] Applying config brokerClientAuthenticationPlugin =
org.apache.pulsar.client.impl.auth.AuthenticationToken
[conf/proxy.conf] Applying config brokerServiceURL =
pulsar://pulsar-300-broker:6650
[conf/proxy.conf] Applying config brokerWebServiceURL =
http://pulsar-300-broker:8080
[conf/proxy.conf] Applying config clusterName = pulsar-300
[conf/proxy.conf] Applying config forwardAuthorizationCredentials = true
[conf/proxy.conf] Applying config httpNumThreads = 8
[conf/proxy.conf] Applying config servicePort = 6650
[conf/proxy.conf] Applying config statusFilePath = /pulsar/status
[conf/proxy.conf] Applying config superUserRoles =
admin,broker-admin,proxy-admin
[conf/proxy.conf] Applying config tokenPublicKey =
file:///pulsar/keys/token/public.key
[conf/proxy.conf] Applying config webServicePort = 80
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil
(file:/pulsar/lib/io.netty-netty-common-4.1.77.Final.jar) to constructor
java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of
io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-11-16T17:17:00,431+ [main] INFO
org.apache.pulsar.broker.authentication.AuthenticationService -
[org.apache.pulsar.broker.authentication.AuthenticationProviderToken] has been
loaded.
2022-11-16T17:17:00,804+ [main] INFO
org.apache.pulsar.proxy.extensions.ProxyExtensionsUtils - Searching for
extensions in /pulsar/./proxyextensions
2022-11-16T17:17:00,806+ [main] WARN
org.apache.pulsar.proxy.extensions.ProxyExtensionsUtils - extension directory
not found
2022-11-16T17:17:00,889+ [main] INFO org.eclipse.jetty.util.log -
Logging initialized @2923ms to org.eclipse.jetty.util.log.Slf4jLog
2022-11-16T17:17:01,052+ [main] INFO
org.apache.pulsar.proxy.server.ProxyService - Started Pulsar Proxy at
/0.0.0.0:6650
2022-11-16T17:17:01,375+ [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm
11.0.16+8-post-Ubuntu-0ubuntu120.04
2022-11-16T17:17:01,423+ [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
2022-11-16T17:17:01,424+ [main] INFO org.eclipse.jetty.server.session -
No SessionScavenger set, using defaults
2022-11-16T17:17:01,427+ [main] INFO org.eclipse.jetty.server.session -
node0 Scavenging every 66ms
2022-11-16T17:17:01,446+ [main] INFO
org.eclipse.jetty.server.handler.ContextHandler - Started
o.e.j.s.ServletContextHandler@4349754{/metrics,null,AVAILABLE}
2022-11-16T17:17:02,121+ [main] WARN
org.glassfish.jersey.server.wadl.WadlFeature - JAXBContext implementation could
not be found. WADL feature is disabled.
2022-11-16T17:17:02,500+ [main] INFO
org.eclipse.jetty.server.handler.ContextHandler - Started
o.e.j.s.ServletContextHandler@b967222{/,null,AVAILABLE}
2022-11-16T17:17:02,557+ [main] WARN
org.glassfish.jersey.server.wadl.WadlFeature - JAXBContext implementation could
not be found. WADL feature is disabled.
2022-11-16T17:17:02,661+ [main] INFO
org.eclipse.jetty.server.handler.ContextHandler - Started
o.e.j.s.ServletContextHandler@6a2eea2a{/proxy-stats,null,AVAILABLE}
2022-11-16T17:17:02,711+ [main] INFO
org.eclipse.jetty.server.handler.ContextHandler - Started
o.e.j.s.ServletContextHandler@2bf94401{/admin,null,AVAILABLE}
2022-11-16T17:17:02,713+ [main] INFO
org.eclipse.jetty.server.handler.ContextHandler - Started
o.e.j.s.ServletContextHandler@2532b351{/lookup,null,AVAILABLE}
2022-11-16T17:17:02,722Z
[jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27] error Uncaught
exception in thread main: Failed to start HTTP server on ports [80]
java.io.IOException: Failed to start HTTP server on ports [80]
at
[GitHub] [pulsar] codecov-commenter commented on pull request #18451: [fix][broker] Correctly set byte and message out totals per subscription
codecov-commenter commented on PR #18451: URL: https://github.com/apache/pulsar/pull/18451#issuecomment-1317368651 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18451?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#18451](https://codecov.io/gh/apache/pulsar/pull/18451?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (216ec00) into [master](https://codecov.io/gh/apache/pulsar/commit/fdf86d3b4e66e25936c2e2143bb5809b079ebc0d?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (fdf86d3) will **decrease** coverage by `11.32%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/pulsar/pull/18451?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #18451 +/- ## = - Coverage 45.67% 34.34% -11.33% + Complexity10075 6458 -3617 = Files 693 617 -76 Lines 6794058477 -9463 Branches 7273 6090 -1183 = - Hits 3103020086-10944 - Misses335759 +2426 + Partials 3577 2632 -945 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `34.34% <0.00%> (-11.33%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18451?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ker/stats/prometheus/NamespaceStatsAggregator.java](https://codecov.io/gh/apache/pulsar/pull/18451/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9wcm9tZXRoZXVzL05hbWVzcGFjZVN0YXRzQWdncmVnYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | | | [...oker/stats/prometheus/PrometheusMetricStreams.java](https://codecov.io/gh/apache/pulsar/pull/18451/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9wcm9tZXRoZXVzL1Byb21ldGhldXNNZXRyaWNTdHJlYW1zLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/pulsar/broker/admin/v1/Brokers.java](https://codecov.io/gh/apache/pulsar/pull/18451/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9Ccm9rZXJzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/pulsar/broker/admin/v1/Clusters.java](https://codecov.io/gh/apache/pulsar/pull/18451/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9DbHVzdGVycy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/pulsar/broker/admin/v1/Properties.java](https://codecov.io/gh/apache/pulsar/pull/18451/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9Qcm9wZXJ0aWVzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/pulsar/broker/admin/v2/ResourceGroups.java](https://codecov.io/gh/apache/pulsar/pull/18451/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9SZXNvdXJjZUdyb3Vwcy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18491: [fix][client] Fix multi-topic consumer stuck after redeliver messages
poorbarcode commented on code in PR #18491:
URL: https://github.com/apache/pulsar/pull/18491#discussion_r1024231244
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##
@@ -712,8 +712,8 @@ public void redeliverUnacknowledgedMessages() {
});
clearIncomingMessages();
unAckedMessageTracker.clear();
+resumeReceivingFromPausedConsumersIfNeeded();
Review Comment:
For multi-topic consumer, there is a lot of optimization to be done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18491: [fix][client] Fix multi-topic consumer stuck after redeliver messages
poorbarcode commented on code in PR #18491:
URL: https://github.com/apache/pulsar/pull/18491#discussion_r1024228893
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##
@@ -712,8 +712,8 @@ public void redeliverUnacknowledgedMessages() {
});
clearIncomingMessages();
unAckedMessageTracker.clear();
+resumeReceivingFromPausedConsumersIfNeeded();
Review Comment:
I think it's possible, but not in this PR. A separate PR needs to be set up
to optimize task scheduling, and I am happy to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #18491: [fix][client] Fix multi-topic consumer stuck after redeliver messages
poorbarcode commented on PR #18491: URL: https://github.com/apache/pulsar/pull/18491#issuecomment-1317283136 Hi @BewareMyPower > > this leads to a higher probability of problems. > Could you explain why? The reason in Menu-Motivation > Or is there a way to make testFailoverConsumerBatchCumulateAck more likely to fail in local env? This test is hard to fail in local env but easy to fail in CI. I guess that in CI the incoming queue can be filled quickly enough before the receive has not started. We can't make it runs faster locally, but we can make receive execution slower, you can change the test like this which makes the test fail in the local env easily: https://user-images.githubusercontent.com/25195800/202235145-e6a4bf17-e792-4cf1-a81a-ac31d024be32.png;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] crossoverJie opened a new pull request, #18505: [refactor][broker] Optimize semantics
crossoverJie opened a new pull request, #18505: URL: https://github.com/apache/pulsar/pull/18505 ### Motivation Make the code easier to read ### Modifications `!isPresent()` -> `isEmpty()` ### Verifying this change - [ ] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun opened a new pull request, #18504: [improve][ci] Allow [fix][broker][branch-2.9] alike patten
tisonkun opened a new pull request, #18504: URL: https://github.com/apache/pulsar/pull/18504 ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] cbornet commented on a diff in pull request #18344: [fix][client] Add support for loading non-serializable properties with ConsumerBuilder::loadConf
cbornet commented on code in PR #18344:
URL: https://github.com/apache/pulsar/pull/18344#discussion_r1024161615
##
pulsar-client/src/test/java/org/apache/pulsar/client/impl/ConsumerBuilderImplTest.java:
##
@@ -370,4 +379,102 @@ public void testTopicConsumerBuilder() {
assertThat(topicConsumerConfigurationData.getTopicNameMatcher().matches("foo")).isTrue();
assertThat(topicConsumerConfigurationData.getPriorityLevel()).isEqualTo(1);
}
+
+@Test
+public void test() {
+Map config = new HashMap<>();
+config.put("keySharedPolicy", KeySharedPolicy.autoSplitHashRange());
+consumerBuilderImpl.loadConf(config);
+}
+
+@Test
+public void testLoadConf() {
+ConsumerBuilderImpl consumerBuilder = new ConsumerBuilderImpl(null,
Schema.BYTES);
+consumerBuilder
+.messageListener((consumer, message) -> {})
+.consumerEventListener(mock(ConsumerEventListener.class))
+
.negativeAckRedeliveryBackoff(MultiplierRedeliveryBackoff.builder().build())
+
.ackTimeoutRedeliveryBackoff(MultiplierRedeliveryBackoff.builder().build())
+.cryptoKeyReader(DefaultCryptoKeyReader.builder().build())
+.messageCrypto(new MessageCryptoBc("ctx1", true))
+
.batchReceivePolicy(BatchReceivePolicy.builder().maxNumBytes(1).build())
+//.keySharedPolicy(mock(KeySharedPolicy.class))
+.messagePayloadProcessor(mock(MessagePayloadProcessor.class));
+
+Map conf = new HashMap<>();
+MessageListener messageListener = (consumer, message) -> {};
+conf.put("messageListener", messageListener);
+ConsumerEventListener consumerEventListener =
mock(ConsumerEventListener.class);
+conf.put("consumerEventListener", consumerEventListener);
+RedeliveryBackoff negativeAckRedeliveryBackoff =
MultiplierRedeliveryBackoff.builder().build();
+conf.put("negativeAckRedeliveryBackoff", negativeAckRedeliveryBackoff);
+RedeliveryBackoff ackTimeoutRedeliveryBackoff =
MultiplierRedeliveryBackoff.builder().build();;
+conf.put("ackTimeoutRedeliveryBackoff", ackTimeoutRedeliveryBackoff);
+CryptoKeyReader cryptoKeyReader =
DefaultCryptoKeyReader.builder().build();
+conf.put("cryptoKeyReader", cryptoKeyReader);
+MessageCrypto messageCrypto = new MessageCryptoBc("ctx2", true);
+conf.put("messageCrypto", messageCrypto);
+BatchReceivePolicy batchReceivePolicy =
BatchReceivePolicy.builder().maxNumBytes(2).build();
+conf.put("batchReceivePolicy", batchReceivePolicy);
+//KeySharedPolicy keySharedPolicy = mock(KeySharedPolicy.class);
Review Comment:
Rebased after fixing #18345
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #18491: [fix][client] Fix multi-topic consumer stuck after redeliver messages
BewareMyPower commented on PR #18491: URL: https://github.com/apache/pulsar/pull/18491#issuecomment-1317201177 > this leads to a higher probability of problems. Could you explain why? Or is there a way to make `testFailoverConsumerBatchCumulateAck` more likely to fail in local env? This test is hard to fail in local env but easy to fail in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] BewareMyPower closed issue #39: [feature request] Batch receive support for Python client
BewareMyPower closed issue #39: [feature request] Batch receive support for Python client URL: https://github.com/apache/pulsar-client-python/issues/39 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] yyj8 commented on pull request #18116: [PIP-214][broker]Add broker level metrics statistics and expose to prometheus
yyj8 commented on PR #18116: URL: https://github.com/apache/pulsar/pull/18116#issuecomment-1317155616 > Don't forget to document the new metrics in a separate PR @yyj8 The new metrics documentation has been completed. For details, please refer to the following PR： https://github.com/apache/pulsar/pull/18503/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #18116: [PIP-214][broker]Add broker level metrics statistics and expose to prometheus
github-actions[bot] commented on PR #18116: URL: https://github.com/apache/pulsar/pull/18116#issuecomment-1317147398 @yyj8 Please select only one documentation label in your PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nodece merged pull request #18498: [improve][test] Print stacktrace when the test fails
nodece merged PR #18498: URL: https://github.com/apache/pulsar/pull/18498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][test] Print stacktrace when the test fails (#18498)
This is an automated email from the ASF dual-hosted git repository.
zixuan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 76308f0bc92 [improve][test] Print stacktrace when the test fails
(#18498)
76308f0bc92 is described below
commit 76308f0bc92d2694e909817fb079f38a0466b722
Author: Zixuan Liu
AuthorDate: Wed Nov 16 22:50:18 2022 +0800
[improve][test] Print stacktrace when the test fails (#18498)
Signed-off-by: Zixuan Liu
---
.../main/java/org/apache/pulsar/tests/PulsarTestListener.java| 9 ++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git
a/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java
b/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java
index 057bad25a0a..b3d70621843 100644
--- a/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java
+++ b/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java
@@ -45,9 +45,12 @@ public class PulsarTestListener implements ITestListener {
System.out.format("! FAILURE-- %s.%s(%s)---\n",
result.getTestClass(),
result.getMethod().getMethodName(),
Arrays.toString(result.getParameters()));
}
-if (result.getThrowable() instanceof ThreadTimeoutException) {
-System.out.println("== THREAD DUMPS ==");
-System.out.println(ThreadDumpUtil.buildThreadDiagnosticString());
+if (result.getThrowable() != null) {
+result.getThrowable().printStackTrace();
+if (result.getThrowable() instanceof ThreadTimeoutException) {
+System.out.println("== THREAD DUMPS ==");
+
System.out.println(ThreadDumpUtil.buildThreadDiagnosticString());
+}
}
}
