[GitHub] [pulsar] PierreZ commented on issue #5735: Provide raw counters metrics

[GitBox]

PierreZ commented on issue #5735: Provide raw counters metrics
URL: https://github.com/apache/pulsar/issues/5735#issuecomment-561520933
 
 
   Thanks, I will do this  


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on a change in pull request #5794: [release][website] Add release notes for Release 2.4.2

[GitBox]

codelipenghui commented on a change in pull request #5794: [release][website] 
Add release notes for Release 2.4.2
URL: https://github.com/apache/pulsar/pull/5794#discussion_r353585946
 
 

 ##
 File path: site2/website/release-notes.md
 ##
 @@ -1,6 +1,109 @@
 
 ## Apache
 
+### 2.4.2  2019-12-04 
+
+ Fixes
+
+* Fixed don't set interrupt flag again after catching interrupt exception in 
Pulsar Client #5643
+* Fixed data is not deleted after expiration due to connected readers #5621
+* Fixed typo in BrokersBase file #5596
+* Fixed the go client docs missing in website #5595
+* Fixed schema def build error with protobuf schema #5569
+* Fixed docs about reset cursor #5551
+* Fixed repeated initialization of connectorsManager #5545
+* Fixed Functions unnecessarily restart during FunctionRuntimeManager init 
phase #5527
+* Fixed list non-persistent topics shows the persistent topics #5502
+* Return after triggering callback with empty result #5500
+* Fixed dispatcher skipping delivery of a batch during concurrent replays #5499
+* Fixed reader_listener option for Python API #5487
+* Fixed wrongly report "3600 messages have timed-out" #5477
+* Fixed broken custom auth-provider that uses authenticationData #5462
+* Fixed negative ack tracker constructor sequence #5453
+* Fixed StringSchema static initialization  #5445
+* Fixed message corruption on OOM for batch messages #5443
+* Fixed couple functions related integration tests #5434
+* Fixed bug that namespace policies does not take effect due to NPE #5408
+* Fixed race condition : Failed to read-more entries on dispatcher #5391
+* Fixed potential deadlock that can occur in addConsumer #5371
+* Fixed proxy to be able to re-send request body #5361
+* Fixed pulsar can't load the customized SerDe #5357
+* Fixed instability in Pulsar Function window integration test #5337
+* Fixed bk write failure part 2 #5322
+* Allow to specify delivery delay in C++ client #5317
 
 Review comment:
   It looks like a enhancement


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on a change in pull request #5794: [release][website] Add release notes for Release 2.4.2

[GitBox]

codelipenghui commented on a change in pull request #5794: [release][website] 
Add release notes for Release 2.4.2
URL: https://github.com/apache/pulsar/pull/5794#discussion_r353586841
 
 

 ##
 File path: site2/website/release-notes.md
 ##
 @@ -1,6 +1,109 @@
 
 ## Apache
 
+### 2.4.2  2019-12-04 
+
+ Fixes
+
+* Fixed don't set interrupt flag again after catching interrupt exception in 
Pulsar Client #5643
+* Fixed data is not deleted after expiration due to connected readers #5621
+* Fixed typo in BrokersBase file #5596
+* Fixed the go client docs missing in website #5595
+* Fixed schema def build error with protobuf schema #5569
+* Fixed docs about reset cursor #5551
+* Fixed repeated initialization of connectorsManager #5545
+* Fixed Functions unnecessarily restart during FunctionRuntimeManager init 
phase #5527
+* Fixed list non-persistent topics shows the persistent topics #5502
+* Return after triggering callback with empty result #5500
+* Fixed dispatcher skipping delivery of a batch during concurrent replays #5499
+* Fixed reader_listener option for Python API #5487
+* Fixed wrongly report "3600 messages have timed-out" #5477
+* Fixed broken custom auth-provider that uses authenticationData #5462
+* Fixed negative ack tracker constructor sequence #5453
+* Fixed StringSchema static initialization  #5445
+* Fixed message corruption on OOM for batch messages #5443
+* Fixed couple functions related integration tests #5434
+* Fixed bug that namespace policies does not take effect due to NPE #5408
+* Fixed race condition : Failed to read-more entries on dispatcher #5391
+* Fixed potential deadlock that can occur in addConsumer #5371
+* Fixed proxy to be able to re-send request body #5361
+* Fixed pulsar can't load the customized SerDe #5357
+* Fixed instability in Pulsar Function window integration test #5337
+* Fixed bk write failure part 2 #5322
+* Allow to specify delivery delay in C++ client #5317
+* Fixed bad_weak_ptr error when closing producer #5315
+* Fixed typo in prometheus metrics #5299
+* Fixed invalidate cache on zk-cache timeout #5298
+* Don't attempt to append on read-only cursor ledger #5297
+* Fixed memory leak caused by not being executed ClientConnection destructor 
#5286
+* Fixed producer blocked after send an over size message while batch enabled 
#5282
+* Fixed race condition while triggering message redelivery after an 
ack-timeout event #5276
+* Fixed behavior when getting a key from functions state that doesn't exist 
#5272
+* Fixed Cmake to build _pulsar for osx #5263
+* Fixed client backoff #5261
+* Fixed memory leak caused by deadline_timer holding object reference #5246
+* Fixed in Message Deduplication that may cause incorrect client/broker 
interaction #5243
+* Fixed bug that fails to search namespace bundle due to NPE #5191
+* Fixed bug that message delivery stops after resetting cursor for failover 
subscription #5185
+* Fixed exception type check order bug #5174
+* Fixed spark receiver to account for all the consumer config options #5152
+* Fixed broker fails to start with function worker enabled and broker client 
using TLS #5151
+* Fixed deadlock when resetting cursor #5139
+* Fixed windowed functions were broken when we changed java function instance 
to use classloaders #5117
+* Fixed storage size always 0 without subscription #5108
+
+ Enhancements
+
+* Add subscribe position param for consumer of sink #5532
+* Efficiency improvements for delay delivery tracker #5498
+* Add is_read_compacted to create_reader() in python API #5483
+* Make some common use method of ManagedLedger public #5472
+* Print unknown exception stacktrace #5444
+* Avoid leak on publish failure on batch message #5442
+* Record message failure and avoid exiting from process on publish failure 
#5441
+* Add support for partitioned topic consumer seek by time #5435
+* Add default loader for latest pyyaml #5432
+* Trim messages which less than mark delete position for message redelivery 
#5378
+* Make skip all messages async #5375
+* Set default ensemble size to 2 in service conf, to match broker.conf #5359
+* Only seek when reading unexpected entry #5356
+* Don't require both region and endpoint to be specified #5355
+* If cursor is not durable, close dispatcher when all consumers are removed 
from subscription #5340
+* Disable stickyRead by default #5321
+* Add debug log + fix thread-factory name  #5302
+* Close previous dispatcher when subscription type changes #5288
+* Improve error handling logic for effectively once #5271
+* Upgrade dependencies for security fixes #5232
+* Allow for topic deletions with regex consumers #5230
+* Ensure consumer background tasks are cancelled after subscribe failures #5204
+* Added missing enum value KeyShared on the python wrapper #5196
+* Make some member variables of Dispatcher volatile #5193
+* Ensure getting list of topics for namespace is handled asynchronously #5188
+* Close RateLimiter instance #5155
+* Throw an error if the key 

[GitHub] [pulsar] codelipenghui commented on a change in pull request #5794: [release][website] Add release notes for Release 2.4.2

[GitBox]

codelipenghui commented on a change in pull request #5794: [release][website] 
Add release notes for Release 2.4.2
URL: https://github.com/apache/pulsar/pull/5794#discussion_r353586287
 
 

 ##
 File path: site2/website/release-notes.md
 ##
 @@ -1,6 +1,109 @@
 
 ## Apache
 
+### 2.4.2  2019-12-04 
+
+ Fixes
+
+* Fixed don't set interrupt flag again after catching interrupt exception in 
Pulsar Client #5643
+* Fixed data is not deleted after expiration due to connected readers #5621
+* Fixed typo in BrokersBase file #5596
+* Fixed the go client docs missing in website #5595
+* Fixed schema def build error with protobuf schema #5569
+* Fixed docs about reset cursor #5551
+* Fixed repeated initialization of connectorsManager #5545
+* Fixed Functions unnecessarily restart during FunctionRuntimeManager init 
phase #5527
+* Fixed list non-persistent topics shows the persistent topics #5502
+* Return after triggering callback with empty result #5500
+* Fixed dispatcher skipping delivery of a batch during concurrent replays #5499
+* Fixed reader_listener option for Python API #5487
+* Fixed wrongly report "3600 messages have timed-out" #5477
+* Fixed broken custom auth-provider that uses authenticationData #5462
+* Fixed negative ack tracker constructor sequence #5453
+* Fixed StringSchema static initialization  #5445
+* Fixed message corruption on OOM for batch messages #5443
+* Fixed couple functions related integration tests #5434
+* Fixed bug that namespace policies does not take effect due to NPE #5408
+* Fixed race condition : Failed to read-more entries on dispatcher #5391
+* Fixed potential deadlock that can occur in addConsumer #5371
+* Fixed proxy to be able to re-send request body #5361
+* Fixed pulsar can't load the customized SerDe #5357
+* Fixed instability in Pulsar Function window integration test #5337
+* Fixed bk write failure part 2 #5322
+* Allow to specify delivery delay in C++ client #5317
+* Fixed bad_weak_ptr error when closing producer #5315
+* Fixed typo in prometheus metrics #5299
+* Fixed invalidate cache on zk-cache timeout #5298
+* Don't attempt to append on read-only cursor ledger #5297
+* Fixed memory leak caused by not being executed ClientConnection destructor 
#5286
+* Fixed producer blocked after send an over size message while batch enabled 
#5282
+* Fixed race condition while triggering message redelivery after an 
ack-timeout event #5276
+* Fixed behavior when getting a key from functions state that doesn't exist 
#5272
+* Fixed Cmake to build _pulsar for osx #5263
+* Fixed client backoff #5261
+* Fixed memory leak caused by deadline_timer holding object reference #5246
+* Fixed in Message Deduplication that may cause incorrect client/broker 
interaction #5243
+* Fixed bug that fails to search namespace bundle due to NPE #5191
+* Fixed bug that message delivery stops after resetting cursor for failover 
subscription #5185
+* Fixed exception type check order bug #5174
+* Fixed spark receiver to account for all the consumer config options #5152
+* Fixed broker fails to start with function worker enabled and broker client 
using TLS #5151
+* Fixed deadlock when resetting cursor #5139
+* Fixed windowed functions were broken when we changed java function instance 
to use classloaders #5117
+* Fixed storage size always 0 without subscription #5108
+
+ Enhancements
+
+* Add subscribe position param for consumer of sink #5532
+* Efficiency improvements for delay delivery tracker #5498
+* Add is_read_compacted to create_reader() in python API #5483
+* Make some common use method of ManagedLedger public #5472
+* Print unknown exception stacktrace #5444
 
 Review comment:
   I think can exclude this one, or describe it in detail


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on a change in pull request #5794: [release][website] Add release notes for Release 2.4.2

[GitBox]

codelipenghui commented on a change in pull request #5794: [release][website] 
Add release notes for Release 2.4.2
URL: https://github.com/apache/pulsar/pull/5794#discussion_r353585013
 
 

 ##
 File path: site2/website/release-notes.md
 ##
 @@ -1,6 +1,109 @@
 
 ## Apache
 
+### 2.4.2  2019-12-04 
+
+ Fixes
+
+* Fixed don't set interrupt flag again after catching interrupt exception in 
Pulsar Client #5643
 
 Review comment:
   Please ensure #5643 can link to the PR


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on issue #5790: [Issue:5687] Add prefix for new keys from Env

[GitBox]

wolfstudy commented on issue #5790: [Issue:5687] Add prefix for new keys from 
Env
URL: https://github.com/apache/pulsar/pull/5790#issuecomment-561511425
 
 
   @cckellogg fixed done, remove `PULSAR_PREFIX_` from `PULSAR_MEM`, 
`PULSAR_GC` and `PULSAR_EXTRA_OPTS `.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy edited a comment on issue #5790: [Issue:5687] Add prefix for new keys from Env

[GitBox]

wolfstudy edited a comment on issue #5790: [Issue:5687] Add prefix for new keys 
from Env
URL: https://github.com/apache/pulsar/pull/5790#issuecomment-561511425
 
 
   @cckellogg fixed done, remove `PULSAR_PREFIX_` from `PULSAR_MEM`, 
`PULSAR_GC` and `PULSAR_EXTRA_OPTS `. PTAL again, thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on a change in pull request #5570: Transaction log implemention

[GitBox]

codelipenghui commented on a change in pull request #5570: Transaction log 
implemention
URL: https://github.com/apache/pulsar/pull/5570#discussion_r353572944
 
 

 ##
 File path: 
pulsar-transaction/coordinator/src/main/java/org/apache/pulsar/transaction/coordinator/exceptions/TransactionMetadataStoreStateException.java
 ##
 @@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.transaction.coordinator.exceptions;
+
+import org.apache.pulsar.transaction.coordinator.TransactionCoordinatorID;
+import 
org.apache.pulsar.transaction.coordinator.TransactionMetadataStoreState.State;
+import org.apache.pulsar.transaction.impl.common.TxnID;
+
+/**
+ * Exception is thrown when a operation of transaction is executed in a error 
transaction metadata store state.
+ */
+public class TransactionMetadataStoreStateException extends 
CoordinatorException {
 
 Review comment:
   It is TransactionStateException not MetadataStoreStateException, right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on a change in pull request #5570: Transaction log implemention

[GitBox]

codelipenghui commented on a change in pull request #5570: Transaction log 
implemention
URL: https://github.com/apache/pulsar/pull/5570#discussion_r353572089
 
 

 ##
 File path: 
pulsar-transaction/coordinator/src/main/java/org/apache/pulsar/transaction/coordinator/TransactionLog.java
 ##
 @@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.transaction.coordinator;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.pulsar.common.api.proto.PulsarApi;
+import org.apache.pulsar.transaction.impl.common.TxnID;
+
+/**
+ * A log interface for transaction to read and write transaction operation.
+ */
+public interface TransactionLog {
+
+
+/**
+ * Replay transaction log to load the transaction map.
+ *
+ * @param txnMetaMap the map of transaction metadata store to load
+ * @param sequenceId the sequence id for transaction metadata store to 
create new transaction
+ * @param replayCallback the call back for replaying the transaction log
+ */
+void replayAsync(ConcurrentMap txnMetaMap,
+AtomicLong sequenceId, ReplayCallback replayCallback);
+
+/**
+ * Read the entry from bookkeeper.
+ *
+ * @param numberOfEntriesToRead the number of reading entry
+ * @param callback the callback to executing when reading entry async 
finished
+ */
+void readAsync(int numberOfEntriesToRead, 
AsyncCallbacks.ReadEntriesCallback callback, Object ctx);
 
 Review comment:
   Both write read and replay should use TransactionMetadataEntry not Entry or 
some other types.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on issue #5795: Update website for 2.4.2 release

[GitBox]

wolfstudy commented on issue #5795: Update website for 2.4.2 release
URL: https://github.com/apache/pulsar/pull/5795#issuecomment-561504274
 
 
   @sijie @jiazhai @codelipenghui @merlimat PTAL, thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy opened a new pull request #5795: Update website for 2.4.2 release

[GitBox]

wolfstudy opened a new pull request #5795: Update website for 2.4.2 release
URL: https://github.com/apache/pulsar/pull/5795
 
 
   Signed-off-by: xiaolong.ran 
   
   ### Modifications
   
   - Update website for 2.4.2 release


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on issue #5794: [release][website] Add release notes for Release 2.4.2

[GitBox]

wolfstudy commented on issue #5794: [release][website] Add release notes for 
Release 2.4.2
URL: https://github.com/apache/pulsar/pull/5794#issuecomment-561499297
 
 
   @sijie @jiazhai @codelipenghui @merlimat PTAL, thanks


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy opened a new pull request #5794: [release][website] Add release notes for Release 2.4.2

[GitBox]

wolfstudy opened a new pull request #5794: [release][website] Add release notes 
for Release 2.4.2
URL: https://github.com/apache/pulsar/pull/5794
 
 
   Signed-off-by: xiaolong.ran 
   
   ### Modifications
   
   - Add release notes for Release 2.4.2.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] congbobo184 commented on issue #5782: Modify the logical in producer carry a schema to connect broker with isAllowAutoUpdateSchema.

[GitBox]

congbobo184 commented on issue #5782: Modify the logical in producer carry a 
schema to connect broker with isAllowAutoUpdateSchema. 
URL: https://github.com/apache/pulsar/pull/5782#issuecomment-561492611
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

svn commit: r37076 - /dev/pulsar/pulsar-2.4.2-candidate-3/ /release/pulsar/pulsar-2.4.2/

[zhaijia]

Author: zhaijia
Date: Wed Dec  4 05:12:32 2019
New Revision: 37076

Log:
xiaolong: pulsar 2.4.2 release

Added:
release/pulsar/pulsar-2.4.2/
  - copied from r37075, dev/pulsar/pulsar-2.4.2-candidate-3/
Removed:
dev/pulsar/pulsar-2.4.2-candidate-3/

[pulsar] annotated tag v2.4.2 updated (811dd45 -> 6b90a31)

[rxl]

This is an automated email from the ASF dual-hosted git repository.

rxl pushed a change to annotated tag v2.4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


*** WARNING: tag v2.4.2 was modified! ***

from 811dd45  (commit)
  to 6b90a31  (tag)
 tagging 811dd45e87600d7163c8f74b40d377831f64bbfb (commit)
 replaces v2.4.2-candidate-1
  by xiaolong.ran
  on Wed Dec 4 13:08:48 2019 +0800

- Log -
Release v2.4.2
-BEGIN PGP SIGNATURE-

iQJDBAABCAAtFiEEHqpd6S06DEdJ6YRttJ34KK/JpEIFAl3nP2APHHJ4bEBhcGFj
aGUub3JnAAoJELSd+CivyaRCMrcQAJXru1m2tcPdtHNY9l26UDcpWNyFT8gbe17h
OKYiL2y2FbLm+uuYE+YZQ9tE426ol0W2psTbic1k0LKDd+7WcWHehoQs/D1rVHeZ
gfOfiI7ENDbJyO24l++SVxBDLUPQLMH3vcsxyhEGtfdEcN3OiDImRExesc3eh9Mk
Rr8eEo9x71RJO4+Eqom+DFeVljgYbKPc+yNlG0Q/u0VUgqZ+WSxsy8DGRdnyFH7s
ilij4a+u+2zXZJpUKibuB2otV6rPbiZkj7fhVqv9uCrcIMyg1m/mj+VioY3nzq+v
z97SkRCdMp1rrWXai/5V3+mey9e7iQy+u4zSx75jSpoLjLCc2Ip/ITSH7C56UmpZ
ZUg4DDHgxd/K5Z4DDm6FTbfCtgR7jlbyIlY05MPEPmfYUS7CrnWHeDcFcY4I7p8t
x3rNukXx//Dpdtu7egJBo5Pqqb0Kqe0z/IJ8ul10oCJVZaHlau3V6lXowDOo1tqJ
ZK3sR4md9l5XPZqiJOagdG+ZOjF9S5s1hYZymWoD359XuFkD38m/nATEb3LwbUut
JpAMrVfzRq1SjxUlZMqkvnrNh52K0ri4f/rsbXy0nU7DBr+orX7tQxAq56Uct8j8
cNufWqVHKLTei7UlONrkQ7FZF4QMM7wldoIrzaEwHEWm7Q189BlbclMGK5g0KkhV
Czlur62d
=hCfr
-END PGP SIGNATURE-
---


No new revisions were added by this update.

Summary of changes:

[GitHub] [pulsar] jiazhai commented on issue #5792: Extend autoSkipNonRecoverableData to apply to missing Schemas in Bookkeeper

[GitBox]

jiazhai commented on issue #5792: Extend autoSkipNonRecoverableData to apply to 
missing Schemas in Bookkeeper
URL: https://github.com/apache/pulsar/issues/5792#issuecomment-561478596
 
 
   Thanks @wmccarley for such a great github issue reporting. 
   @wmccarley Would you like to contribute a fix for this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on a change in pull request #5790: [Issue:5687] Add prefix for new keys from Env

[GitBox]

wolfstudy commented on a change in pull request #5790: [Issue:5687] Add prefix 
for new keys from Env
URL: https://github.com/apache/pulsar/pull/5790#discussion_r353541860
 
 

 ##
 File path: deployment/kubernetes/aws/broker.yaml
 ##
 @@ -25,15 +25,15 @@ metadata:
 data:
 # Tune for available memory. Increase the heap up to 24G to have
 # better GC behavior at high throughput
-PULSAR_MEM: "\" -Dio.netty.leakDetectionLevel=disabled 
-Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled 
-XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis 
-XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 
-XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError 
-XX:+PerfDisableSharedMem -Xms12g -Xmx12g -XX:MaxDirectMemorySize=14g 
-Dpulsar.root.logger=DEBUG,FILE \""
-PULSAR_GC: "\" -XX:+UseG1GC -XX:MaxGCPauseMillis=10\""
-zookeeperServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
-configurationStoreServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
-clusterName: us-east
-managedLedgerDefaultEnsembleSize: "2"
-managedLedgerDefaultWriteQuorum: "2"
-managedLedgerDefaultAckQuorum: "2"
-deduplicationEnabled: "false"
+PULSAR_PREFIX_PULSAR_MEM: "\" -Dio.netty.leakDetectionLevel=disabled 
-Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled 
-XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis 
-XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 
-XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError 
-XX:+PerfDisableSharedMem -Xms12g -Xmx12g -XX:MaxDirectMemorySize=14g 
-Dpulsar.root.logger=DEBUG,FILE \""
 
 Review comment:
   > There is no need to put a prefix in front of PULSAR_MEM, PULSAR_GC, or 
PULSAR_EXTRA_OPTS. This script 
(https://github.com/apache/pulsar/blob/master/conf/pulsar_env.sh) will pick up 
the environment variables. Same for all the other places.
   
   ok, I will fix them.
   
   > Every PULSAR_PREFX_ key value will end up in the conf/pulsar_env.sh 
script. That might not matter now or break things but I don't think is the 
correct approach in general for the broker or proxy configs.
   
   Yes, this change is only used to fix problems encountered by users during 
the current use.
   
   >In my opinion the override scripts should be completely removed from broker 
and proxy and function workers. There are ways to apply configs without doing 
search and replaces with scripts and i think this is cleaner and less error 
prone.
   
   The proposal looks good to me,  this is a big change, maybe we can open a 
PIP to discuss further details.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] kaykillyou commented on issue #5668: bookie server's log error: Failed to restore rocksdb 000000000000000000/000000000000000001/000000000000000000

[GitBox]

kaykillyou commented on issue #5668: bookie server's log error: Failed to 
restore rocksdb 00/01/00
URL: https://github.com/apache/pulsar/issues/5668#issuecomment-561465598
 
 
   @jiazhai OK, look forward to the next version update.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] kaykillyou closed issue #5668: bookie server's log error: Failed to restore rocksdb 000000000000000000/000000000000000001/000000000000000000

[GitBox]

kaykillyou closed issue #5668: bookie server's log error: Failed to restore 
rocksdb 00/01/00
URL: https://github.com/apache/pulsar/issues/5668
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on issue #5781: [Issue:4753] Add encryption docs to Functions develop

[GitBox]

wolfstudy commented on issue #5781: [Issue:4753] Add encryption docs to 
Functions develop
URL: https://github.com/apache/pulsar/pull/5781#issuecomment-561453345
 
 
   > I don't think that's what the original github issue asks for. The original 
issue asks for the documentation around security especially secrets 
(https://github.com/apache/pulsar/tree/master/pulsar-functions/secrets) in the 
development of functions. It is not about security in the function worker.
   
   Thanks @sijie clarification, will fix it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] congbobo184 commented on issue #5680: [transaction-coordinator] Implementation of transaction coordinator client.

[GitBox]

congbobo184 commented on issue #5680: [transaction-coordinator] Implementation 
of transaction coordinator client.
URL: https://github.com/apache/pulsar/pull/5680#issuecomment-561453917
 
 
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] congbobo184 commented on issue #5680: [transaction-coordinator] Implementation of transaction coordinator client.

[GitBox]

congbobo184 commented on issue #5680: [transaction-coordinator] Implementation 
of transaction coordinator client.
URL: https://github.com/apache/pulsar/pull/5680#issuecomment-561453776
 
 
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] congbobo184 removed a comment on issue #5680: [transaction-coordinator] Implementation of transaction coordinator client.

[GitBox]

congbobo184 removed a comment on issue #5680: [transaction-coordinator] 
Implementation of transaction coordinator client.
URL: https://github.com/apache/pulsar/pull/5680#issuecomment-561453701
 
 
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] congbobo184 removed a comment on issue #5680: [transaction-coordinator] Implementation of transaction coordinator client.

[GitBox]

congbobo184 removed a comment on issue #5680: [transaction-coordinator] 
Implementation of transaction coordinator client.
URL: https://github.com/apache/pulsar/pull/5680#issuecomment-561453776
 
 
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] congbobo184 commented on issue #5680: [transaction-coordinator] Implementation of transaction coordinator client.

[GitBox]

congbobo184 commented on issue #5680: [transaction-coordinator] Implementation 
of transaction coordinator client.
URL: https://github.com/apache/pulsar/pull/5680#issuecomment-561453701
 
 
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on a change in pull request #5782: Modify the logical in producer carry a schema to connect broker with isAllowAutoUpdateSchema.

[GitBox]

codelipenghui commented on a change in pull request #5782: Modify the logical 
in producer carry a schema to connect broker with isAllowAutoUpdateSchema. 
URL: https://github.com/apache/pulsar/pull/5782#discussion_r353522311
 
 

 ##
 File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
 ##
 @@ -185,13 +185,19 @@ public String getReplicatorPrefix() {
 
 String base = TopicName.get(getName()).getPartitionedTopicName();
 String id = TopicName.get(base).getSchemaName();
-if (isAllowAutoUpdateSchema) {
-return brokerService.pulsar()
-.getSchemaRegistryService()
-.putSchemaIfAbsent(id, schema, 
schemaCompatibilityStrategy);
-} else {
-return FutureUtil.failedFuture(new 
IncompatibleSchemaException("Don't allow auto update schema."));
-}
+SchemaRegistryService schemaRegistryService = 
brokerService.pulsar().getSchemaRegistryService();
+return isAllowAutoUpdateSchema ? schemaRegistryService
+.putSchemaIfAbsent(id, schema, schemaCompatibilityStrategy)
+: 
schemaRegistryService.trimDeletedSchemaAndGetList(id).thenCompose(schemaAndMetadataList
 ->
+
schemaRegistryService.getSchemaVersionBySchemaData(schemaAndMetadataList, 
schema)
+.thenCompose(schemaVersion -> {
+if (schemaVersion == null) {
+return FutureUtil
+.failedFuture(new 
IncompatibleSchemaException("Don't allow auto update schema."));
 
 Review comment:
   ```suggestion
   .failedFuture(new 
IncompatibleSchemaException("Schema not found and schema auto updating is 
disabled"));
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on a change in pull request #5782: Modify the logical in producer carry a schema to connect broker with isAllowAutoUpdateSchema.

[GitBox]

codelipenghui commented on a change in pull request #5782: Modify the logical 
in producer carry a schema to connect broker with isAllowAutoUpdateSchema. 
URL: https://github.com/apache/pulsar/pull/5782#discussion_r353524616
 
 

 ##
 File path: 
pulsar-broker/src/test/java/org/apache/pulsar/schema/compatibility/SchemaCompatibilityCheckTest.java
 ##
 @@ -267,6 +269,23 @@ public void 
testIsAutoUpdateSchema(SchemaCompatibilityStrategy schemaCompatibili
 assertEquals(personTwo.id, 2);
 assertEquals(personTwo.name, "Lucy");
 
+producer.close();
+consumerTwo.close();
+
+producer = producerThreeBuilder.create();
+consumerTwo = comsumerBuilder.subscribe();
+
+
admin.namespaces().setIsAllowAutoUpdateSchema(namespaceName.toString(), false);
 
 Review comment:
   Set schema auto update strategy need before producer creation, because we do 
the schema compatibility check when create a new producer or consumer.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on issue #3813: [go client] Pointers vs. values in parameters

[GitBox]

wolfstudy commented on issue #3813: [go client] Pointers vs. values in 
parameters 
URL: https://github.com/apache/pulsar/issues/3813#issuecomment-56144
 
 
   > @wolfstudy did we address the problem in native client implementation?
   
   No, in the native go client, we still have this problem. 
[pr:69](https://github.com/apache/pulsar-client-go/pull/69) is a better way to 
solve this problem.
   
   In cgo client, maybe we can ignore this the problem, make sure sdk is 
forward compatible.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on issue #5782: Modify the logical in producer carry a schema to connect broker with isAllowAutoUpdateSchema.

[GitBox]

codelipenghui commented on issue #5782: Modify the logical in producer carry a 
schema to connect broker with isAllowAutoUpdateSchema. 
URL: https://github.com/apache/pulsar/pull/5782#issuecomment-561446618
 
 
   @sijie Yes, it's better to be include in 2.5.0,  if user disable schema 
auto_update, although producer using an exist schema, broker still will reject 
it. So we need this PR in 2.5.0


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on issue #5417: Broker hangs and crashes when listing non-persistent topics

[GitBox]

tuteng commented on issue #5417: Broker hangs and crashes when listing 
non-persistent topics
URL: https://github.com/apache/pulsar/issues/5417#issuecomment-561446569
 
 
   Thank you for your reply. I will test it under the cluster and try to fix 
this problem.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie commented on a change in pull request #5788: [pulsar-java-client] Decouple partition switch frequency with maxPublishDelay in producer configuration

[GitBox]

sijie commented on a change in pull request #5788: [pulsar-java-client] 
Decouple partition switch frequency with maxPublishDelay in producer 
configuration
URL: https://github.com/apache/pulsar/pull/5788#discussion_r353510004
 
 

 ##
 File path: 
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
 ##
 @@ -349,6 +349,20 @@
  */
 ProducerBuilder batchingMaxPublishDelay(long batchDelay, TimeUnit 
timeUnit);
 
+/**
+ * Set the partition switch frequency while batching of messages is 
enabled and
+ * using round-robin routing mode for non-keyed message default: 10.
+ *
+ * The time period of partition switch is frequency * 
batchingMaxPublishDelay. During this period,
+ * all messages arrives will be route to the same partition.
+ *
+ * @param frequency the frequency of partition switch
+ * @return the producer builder instance
+ * @see #messageRoutingMode(MessageRoutingMode)
+ * @see #batchingMaxPublishDelay(long, TimeUnit)
+ */
+ProducerBuilder batchingPartitionSwitchFrequency(int frequency);
 
 Review comment:
   Oh I misread the code. I see it now.  Then that's fine to me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie commented on issue #5735: Provide raw counters metrics

[GitBox]

sijie commented on issue #5735: Provide raw counters metrics
URL: https://github.com/apache/pulsar/issues/5735#issuecomment-561433734
 
 
   I don't think we have used Prometheus counter directly.
   
   I would suggest:
   
   - use LongAdder in AbstractTopic for counting only.
   - expose those counters in PersistentInternalTopicStats or TopicStats
   - update NamespaceStatsAggregator to expose those counters to prometheus 
servlet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] yjshen commented on a change in pull request #5788: [pulsar-java-client] Decouple partition switch frequency with maxPublishDelay in producer configuration

[GitBox]

yjshen commented on a change in pull request #5788: [pulsar-java-client] 
Decouple partition switch frequency with maxPublishDelay in producer 
configuration
URL: https://github.com/apache/pulsar/pull/5788#discussion_r353508179
 
 

 ##
 File path: 
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
 ##
 @@ -349,6 +349,20 @@
  */
 ProducerBuilder batchingMaxPublishDelay(long batchDelay, TimeUnit 
timeUnit);
 
+/**
+ * Set the partition switch frequency while batching of messages is 
enabled and
+ * using round-robin routing mode for non-keyed message default: 10.
+ *
+ * The time period of partition switch is frequency * 
batchingMaxPublishDelay. During this period,
+ * all messages arrives will be route to the same partition.
+ *
+ * @param frequency the frequency of partition switch
+ * @return the producer builder instance
+ * @see #messageRoutingMode(MessageRoutingMode)
+ * @see #batchingMaxPublishDelay(long, TimeUnit)
+ */
+ProducerBuilder batchingPartitionSwitchFrequency(int frequency);
 
 Review comment:
   The current frequency is used to compute switch delay in `frequency` * 
`maxBatchDelayMicros`. Do you mean I should make it a standalone variable? Then 
if users didn't set this variable, or the user is setting a variable less than 
`maxBatchDelay`, I need to use `maxBatchDelay` in round-robin partitioner?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Jennifer88huang commented on issue #5465: [Pulsar Functions] User class constructor throws exception

[GitBox]

Jennifer88huang commented on issue #5465: [Pulsar Functions] User class 
constructor throws exception
URL: https://github.com/apache/pulsar/issues/5465#issuecomment-561428379
 
 
   Ok, when you fix the logic, if you need any support in writing docs, feel 
free to let me know.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Jennifer88huang edited a comment on issue #5465: [Pulsar Functions] User class constructor throws exception

[GitBox]

Jennifer88huang edited a comment on issue #5465: [Pulsar Functions] User class 
constructor throws exception
URL: https://github.com/apache/pulsar/issues/5465#issuecomment-561428379
 
 
   Ok, when you fix the logic, if you need any support in writing docs, feel 
free to contact me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] zhaohaidao commented on a change in pull request #5767: Support batch authorization of partitioned topic

[GitBox]

zhaohaidao commented on a change in pull request #5767: Support batch 
authorization of partitioned topic
URL: https://github.com/apache/pulsar/pull/5767#discussion_r353499759
 
 

 ##
 File path: 
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java
 ##
 @@ -312,4 +316,76 @@ public void testGetPartitionedTopicsList() throws 
KeeperException, InterruptedEx
 Assert.assertEquals(nonPersistentPartitionedTopics.size(), 1);
 
Assert.assertEquals(TopicName.get(nonPersistentPartitionedTopics.get(0)).getDomain().value(),
 TopicDomain.non_persistent.value());
 }
+
+@Test
+public void testGrantNonPartitionedTopic() {
+final String topicName = "non-partitioned-topic";
+persistentTopics.createNonPartitionedTopic(testTenant, testNamespace, 
topicName, true);
+String role = "role";
+Set expectActions = new HashSet<>();
+expectActions.add(AuthAction.produce);
+persistentTopics.grantPermissionsOnTopic(testTenant, testNamespace, 
topicName, role, expectActions);
+Map> permissions = 
persistentTopics.getPermissionsOnTopic(testTenant, testNamespace, topicName);
+Assert.assertEquals(permissions.get(role), expectActions);
+}
+
+@Test
+public void testGrantPartitionedTopic() {
+final String partitionedTopicName = "partitioned-topic";
+final int numPartitions = 5;
+persistentTopics.createPartitionedTopic(testTenant, testNamespace, 
partitionedTopicName, numPartitions);
+
+String role = "role";
+Set expectActions = new HashSet<>();
+expectActions.add(AuthAction.produce);
+persistentTopics.grantPermissionsOnTopic(testTenant, testNamespace, 
partitionedTopicName, role, expectActions);
+Map> permissions = 
persistentTopics.getPermissionsOnTopic(testTenant, testNamespace,
+partitionedTopicName);
+Assert.assertEquals(permissions.get(role), expectActions);
+TopicName topicName = TopicName.get(TopicDomain.persistent.value(), 
testTenant, testNamespace,
+partitionedTopicName);
+for (int i = 0; i < numPartitions; i++) {
+TopicName partition = topicName.getPartition(i);
+Map> partitionPermissions = 
persistentTopics.getPermissionsOnTopic(testTenant,
+testNamespace, partition.getEncodedLocalName());
 
 Review comment:
   Thanks for reminding. PersistentTopicsTest passed locally.
   I use getEncodedLocalName because the parameter for the rest API 
getPermissionsOnTopic is named encodedTopic as followed.
   ```java
   public Map> getPermissionsOnTopic(
   @ApiParam(value = "Specify the tenant", required = true)
   @PathParam("tenant") String tenant,
   @ApiParam(value = "Specify the namespace", required = true)
   @PathParam("namespace") String namespace,
   @ApiParam(value = "Specify topic name", required = true)
   @PathParam("topic") @Encoded String encodedTopic) {
   validateTopicName(tenant, namespace, encodedTopic);
   return internalGetPermissionsOnTopic();
   }
   ``` 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] zhaohaidao commented on a change in pull request #5767: Support batch authorization of partitioned topic

[GitBox]

zhaohaidao commented on a change in pull request #5767: Support batch 
authorization of partitioned topic
URL: https://github.com/apache/pulsar/pull/5767#discussion_r353499759
 
 

 ##
 File path: 
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java
 ##
 @@ -312,4 +316,76 @@ public void testGetPartitionedTopicsList() throws 
KeeperException, InterruptedEx
 Assert.assertEquals(nonPersistentPartitionedTopics.size(), 1);
 
Assert.assertEquals(TopicName.get(nonPersistentPartitionedTopics.get(0)).getDomain().value(),
 TopicDomain.non_persistent.value());
 }
+
+@Test
+public void testGrantNonPartitionedTopic() {
+final String topicName = "non-partitioned-topic";
+persistentTopics.createNonPartitionedTopic(testTenant, testNamespace, 
topicName, true);
+String role = "role";
+Set expectActions = new HashSet<>();
+expectActions.add(AuthAction.produce);
+persistentTopics.grantPermissionsOnTopic(testTenant, testNamespace, 
topicName, role, expectActions);
+Map> permissions = 
persistentTopics.getPermissionsOnTopic(testTenant, testNamespace, topicName);
+Assert.assertEquals(permissions.get(role), expectActions);
+}
+
+@Test
+public void testGrantPartitionedTopic() {
+final String partitionedTopicName = "partitioned-topic";
+final int numPartitions = 5;
+persistentTopics.createPartitionedTopic(testTenant, testNamespace, 
partitionedTopicName, numPartitions);
+
+String role = "role";
+Set expectActions = new HashSet<>();
+expectActions.add(AuthAction.produce);
+persistentTopics.grantPermissionsOnTopic(testTenant, testNamespace, 
partitionedTopicName, role, expectActions);
+Map> permissions = 
persistentTopics.getPermissionsOnTopic(testTenant, testNamespace,
+partitionedTopicName);
+Assert.assertEquals(permissions.get(role), expectActions);
+TopicName topicName = TopicName.get(TopicDomain.persistent.value(), 
testTenant, testNamespace,
+partitionedTopicName);
+for (int i = 0; i < numPartitions; i++) {
+TopicName partition = topicName.getPartition(i);
+Map> partitionPermissions = 
persistentTopics.getPermissionsOnTopic(testTenant,
+testNamespace, partition.getEncodedLocalName());
 
 Review comment:
   Thanks for reminding. PersistentTopicsTest passed locally.
   I use getEncodedLocalName because the parameter for rest API is named 
encodedTopic as follwed.
   ```java
   public Map> getPermissionsOnTopic(
   @ApiParam(value = "Specify the tenant", required = true)
   @PathParam("tenant") String tenant,
   @ApiParam(value = "Specify the namespace", required = true)
   @PathParam("namespace") String namespace,
   @ApiParam(value = "Specify topic name", required = true)
   @PathParam("topic") @Encoded String encodedTopic) {
   validateTopicName(tenant, namespace, encodedTopic);
   return internalGetPermissionsOnTopic();
   }
   ``` 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] zhaohaidao commented on a change in pull request #5767: Support batch authorization of partitioned topic

[GitBox]

zhaohaidao commented on a change in pull request #5767: Support batch 
authorization of partitioned topic
URL: https://github.com/apache/pulsar/pull/5767#discussion_r353498429
 
 

 ##
 File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
 ##
 @@ -301,13 +295,29 @@ protected void internalGrantPermissionsOnTopic(String 
role, Set acti
 log.warn("[{}] Failed to grant permissions on topic {}: concurrent 
modification", clientAppId(),
 topicUri);
 throw new RestException(Status.CONFLICT, "Concurrent 
modification");
-}
-catch (Exception e) {
+} catch (Exception e) {
 log.error("[{}] Failed to grant permissions for topic {}", 
clientAppId(), topicUri, e);
 throw new RestException(e);
 }
 }
 
+protected void internalGrantPermissionsOnTopic(String role, 
Set actions) {
+// This operation should be reading from zookeeper and it should be 
allowed without having admin privileges
+validateAdminAccessForTenant(namespaceName.getTenant());
+validatePoliciesReadOnlyAccess();
+
+PartitionedTopicMetadata meta = getPartitionedTopicMetadata(topicName, 
true, false);
 
 Review comment:
   Thanks for reminding. I have fixed this failed case.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] lovelle opened a new pull request #5793: Fixes some minor thread safety violations

[GitBox]

lovelle opened a new pull request #5793: Fixes some minor thread safety 
violations
URL: https://github.com/apache/pulsar/pull/5793
 
 
   *Motivation*
   
   Reading latest changes through pulsar code I found some minor thread safety
   violations.
   
   *Modifications*
   
 - Add synchronized method to poll `pendingBatchReceives from a
   non-synchronized method.
 - Protects hasPendingBatchReceive method with synchronized.
 - When possible, prefer using final fields. fixed for initialStartMessageId
   and startMessageRollbackDurationInSec fields.
 - Fix issue accessing timeout field within inner lock on 
UnAckedMessageTracker.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] digi691 commented on issue #5417: Broker hangs and crashes when listing non-persistent topics

[GitBox]

digi691 commented on issue #5417: Broker hangs and crashes when listing 
non-persistent topics
URL: https://github.com/apache/pulsar/issues/5417#issuecomment-561326130
 
 
   What is the appropriate number?  Like I said earlier, I currently have it 
set to 8.  Which is the suggested default.  What I don't understand is that if 
I make any other requests on the API, it finishes fine.  It's only this 
endpoint that causes the whole broker to crash.  I don't have proof of what is 
going on, though it seems like when I hit that endpoint it ties up a 
HttpServerThread indefinitely.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie commented on a change in pull request #5788: [pulsar-java-client] Decouple partition switch frequency with maxPublishDelay in producer configuration

[GitBox]

sijie commented on a change in pull request #5788: [pulsar-java-client] 
Decouple partition switch frequency with maxPublishDelay in producer 
configuration
URL: https://github.com/apache/pulsar/pull/5788#discussion_r35567
 
 

 ##
 File path: 
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
 ##
 @@ -349,6 +349,20 @@
  */
 ProducerBuilder batchingMaxPublishDelay(long batchDelay, TimeUnit 
timeUnit);
 
+/**
+ * Set the partition switch frequency while batching of messages is 
enabled and
+ * using round-robin routing mode for non-keyed message default: 10.
+ *
+ * The time period of partition switch is frequency * 
batchingMaxPublishDelay. During this period,
+ * all messages arrives will be route to the same partition.
+ *
+ * @param frequency the frequency of partition switch
+ * @return the producer builder instance
+ * @see #messageRoutingMode(MessageRoutingMode)
+ * @see #batchingMaxPublishDelay(long, TimeUnit)
+ */
+ProducerBuilder batchingPartitionSwitchFrequency(int frequency);
 
 Review comment:
   It is better to add both time interval and time unit in the method, by 
following the convention of other time related settings.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie merged pull request #5757: [Issue 5562][docs]Modify schema compatibility check doc

[GitBox]

sijie merged pull request #5757: [Issue 5562][docs]Modify schema compatibility 
check doc
URL: https://github.com/apache/pulsar/pull/5757
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie closed issue #5562: Add the doc for new schema compatibility strategy.

[GitBox]

sijie closed issue #5562: Add the doc for new schema compatibility  strategy.
URL: https://github.com/apache/pulsar/issues/5562
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie commented on issue #5757: [Issue 5562][docs]Modify schema compatibility check doc

[GitBox]

sijie commented on issue #5757: [Issue 5562][docs]Modify schema compatibility 
check doc
URL: https://github.com/apache/pulsar/pull/5757#issuecomment-561282299
 
 
   > @congbobo184 @Anonymitaet When you fix an issue, please add the issue 
number and component in the title, for example, "[Issue 5562][docs] xxx", and 
add "Fixes #" in the description.
   > So when the PR is merged, the corresponding issue will be closed 
automatically.
   
   maybe we can add this to the contributor guide and encourage committers and 
contributors following the same practice.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie commented on issue #5782: Modify the logical in producer carry a schema to connect broker with isAllowAutoUpdateSchema.

[GitBox]

sijie commented on issue #5782: Modify the logical in producer carry a schema 
to connect broker with isAllowAutoUpdateSchema. 
URL: https://github.com/apache/pulsar/pull/5782#issuecomment-561281558
 
 
   @codelipenghui is this required for 2.5.0 release?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie commented on issue #5147: Deprecate getstatus subcommand for source command

[GitBox]

sijie commented on issue #5147: Deprecate getstatus subcommand for source 
command
URL: https://github.com/apache/pulsar/pull/5147#issuecomment-561279653
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] dependabot[bot] commented on issue #5638: Bump jackson-databind from 2.9.9.3 to 2.10.1

[GitBox]

dependabot[bot] commented on issue #5638: Bump jackson-databind from 2.9.9.3 to 
2.10.1
URL: https://github.com/apache/pulsar/pull/5638#issuecomment-561279738
 
 
   Looks like com.fasterxml.jackson.core:jackson-databind is up-to-date now, so 
this is no longer needed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] dependabot[bot] closed pull request #5638: Bump jackson-databind from 2.9.9.3 to 2.10.1

[GitBox]

dependabot[bot] closed pull request #5638: Bump jackson-databind from 2.9.9.3 
to 2.10.1
URL: https://github.com/apache/pulsar/pull/5638
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie commented on issue #5776: fix potential NPE and repeated conditional test

[GitBox]

sijie commented on issue #5776: fix potential NPE and repeated conditional test
URL: https://github.com/apache/pulsar/pull/5776#issuecomment-561279110
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie merged pull request #5758: Bump jackson libraries to 2.10.1

[GitBox]

sijie merged pull request #5758: Bump jackson libraries to 2.10.1
URL: https://github.com/apache/pulsar/pull/5758
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[pulsar] branch master updated: Bump jackson libraries to 2.10.1 (#5758)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 9ad9535  Bump jackson libraries to 2.10.1 (#5758)
9ad9535 is described below

commit 9ad95357a9f6005eb38e840424f36e501c885190
Author: Masahiro Sakamoto 
AuthorDate: Wed Dec 4 02:45:20 2019 +0900

Bump jackson libraries to 2.10.1 (#5758)

Updated jackson libraries to the latest version. There is a security 
vulnerability in `jackson-databind` currently used by Pulsar.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17531
---
 distribution/server/licenses/LICENSE-EDL-1.0.txt | 30 
 distribution/server/src/assemble/LICENSE.bin.txt | 22 ++---
 pom.xml  |  4 ++--
 3 files changed, 45 insertions(+), 11 deletions(-)

diff --git a/distribution/server/licenses/LICENSE-EDL-1.0.txt 
b/distribution/server/licenses/LICENSE-EDL-1.0.txt
new file mode 100644
index 000..6118dac
--- /dev/null
+++ b/distribution/server/licenses/LICENSE-EDL-1.0.txt
@@ -0,0 +1,30 @@
+Copyright (c) 2007, Eclipse Foundation, Inc. and its licensors.
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+  - Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+
+  - Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+
+  - Neither the name of the Eclipse Foundation, Inc. nor the names of its
+contributors may be used to endorse or promote products derived
+from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt 
b/distribution/server/src/assemble/LICENSE.bin.txt
index b08c698..470d05c 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -314,14 +314,14 @@ The Apache Software License, Version 2.0
  * Jackson
  - org.codehaus.jackson-jackson-core-asl-1.9.13.jar
  - org.codehaus.jackson-jackson-mapper-asl-1.9.13.jar
- - com.fasterxml.jackson.core-jackson-annotations-2.9.9.jar
- - com.fasterxml.jackson.core-jackson-core-2.9.9.jar
- - com.fasterxml.jackson.core-jackson-databind-2.9.9.3.jar
- - com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.9.9.jar
- - com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.9.9.jar
- - com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.9.9.jar
- - com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.9.9.jar
- - com.fasterxml.jackson.module-jackson-module-jsonSchema-2.9.9.jar
+ - com.fasterxml.jackson.core-jackson-annotations-2.10.1.jar
+ - com.fasterxml.jackson.core-jackson-core-2.10.1.jar
+ - com.fasterxml.jackson.core-jackson-databind-2.10.1.jar
+ - com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.10.1.jar
+ - com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.10.1.jar
+ - com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.10.1.jar
+ - com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.10.1.jar
+ - com.fasterxml.jackson.module-jackson-module-jsonSchema-2.10.1.jar
  * Caffeine -- com.github.ben-manes.caffeine-caffeine-2.6.2.jar
  * Proto Google Common Protos -- 
com.google.api.grpc-proto-google-common-protos-1.12.0.jar
  * Gson -- com.google.code.gson-gson-2.8.2.jar
@@ -429,7 +429,7 @@ The Apache Software License, Version 2.0
 - org.eclipse.jetty.websocket-websocket-common-9.4.20.v20190813.jar
 - org.eclipse.jetty.websocket-websocket-server-9.4.20.v20190813.jar
 - org.eclipse.jetty.websocket-websocket-servlet-9.4.20.v20190813.jar
- * SnakeYaml -- org.yaml-snakeyaml-1.23.jar
+ * SnakeYaml -- org.yaml-snakeyaml-1.24.jar
  * RocksDB - org.rocksdb-rocksdbjni-5.13.3.jar
  * HttpClient
 - 

[GitHub] [pulsar] sijie merged pull request #5785: Modify the logical for getting schema compatibility strategy

[GitBox]

sijie merged pull request #5785: Modify the logical for getting schema 
compatibility strategy
URL: https://github.com/apache/pulsar/pull/5785
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[pulsar] branch master updated: Modify the logical for getting schema compatibility strategy (#5785)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new c952b6b  Modify the logical for getting schema compatibility strategy 
(#5785)
c952b6b is described below

commit c952b6b59ebc5f3e156de916ee9f2f88aa86732c
Author: congbo <39078850+congbobo...@users.noreply.github.com>
AuthorDate: Wed Dec 4 01:39:02 2019 +0800

Modify the logical for getting schema compatibility strategy (#5785)

### Motivation
Modify the logical for getting schema compatibility strategy, if 
schemaCompatibilityStrategy = SchemaCompatibilityStrategy.UNDEFINED return the 
SchemaAutoUpdateCompatibilityStrategy.

### Verifying this change
Add the tests for it
---
 .../java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java  | 8 +++-
 .../pulsar/schema/compatibility/SchemaCompatibilityCheckTest.java | 4 
 .../src/main/java/org/apache/pulsar/admin/cli/CmdNamespaces.java  | 4 ++--
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
index 85d043f..adc7bd4 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
@@ -2044,7 +2044,13 @@ public abstract class NamespacesBase extends 
AdminResource {
 
 protected SchemaCompatibilityStrategy 
internalGetSchemaCompatibilityStrategy() {
 validateAdminAccessForTenant(namespaceName.getTenant());
-return 
getNamespacePolicies(namespaceName).schema_compatibility_strategy;
+Policies policies = getNamespacePolicies(namespaceName);
+SchemaCompatibilityStrategy schemaCompatibilityStrategy = 
policies.schema_compatibility_strategy;
+if (schemaCompatibilityStrategy == 
SchemaCompatibilityStrategy.UNDEFINED){
+schemaCompatibilityStrategy = SchemaCompatibilityStrategy
+
.fromAutoUpdatePolicy(policies.schema_auto_update_compatibility_strategy);
+}
+return schemaCompatibilityStrategy;
 }
 
 @Deprecated
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/schema/compatibility/SchemaCompatibilityCheckTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/schema/compatibility/SchemaCompatibilityCheckTest.java
index f091d9a..65db37e 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/schema/compatibility/SchemaCompatibilityCheckTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/schema/compatibility/SchemaCompatibilityCheckTest.java
@@ -45,6 +45,7 @@ import java.util.concurrent.ThreadLocalRandom;
 
 import static org.apache.pulsar.common.naming.TopicName.PUBLIC_TENANT;
 import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
 
 
 @Slf4j
@@ -233,6 +234,9 @@ public class SchemaCompatibilityCheckTest extends 
MockedPulsarServiceBaseTest {
 Sets.newHashSet(CLUSTER_NAME)
 );
 
+
assertEquals(admin.namespaces().getSchemaCompatibilityStrategy(namespaceName.toString()),
+SchemaCompatibilityStrategy.FULL);
+
 
admin.namespaces().setSchemaCompatibilityStrategy(namespaceName.toString(), 
schemaCompatibilityStrategy);
 admin.schemas().createSchema(fqtn, 
Schema.AVRO(Schemas.PersonOne.class).getSchemaInfo());
 
diff --git 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNamespaces.java
 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNamespaces.java
index eb72052..9dc8699 100644
--- 
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNamespaces.java
+++ 
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNamespaces.java
@@ -1082,7 +1082,7 @@ public class CmdNamespaces extends CmdBase {
 }
 }
 
-@Parameters(commandDescription = "Get the schema auto-update strategy for 
a namespace")
+@Parameters(commandDescription = "Get the schema auto-update strategy for 
a namespace", hidden = true)
 private class GetSchemaAutoUpdateStrategy extends CliCommand {
 @Parameter(description = "tenant/namespace", required = true)
 private java.util.List params;
@@ -1095,7 +1095,7 @@ public class CmdNamespaces extends CmdBase {
 }
 }
 
-@Parameters(commandDescription = "Set the schema auto-update strategy for 
a namespace")
+@Parameters(commandDescription = "Set the schema auto-update strategy for 
a namespace", hidden = true)
 private class SetSchemaAutoUpdateStrategy extends CliCommand {
 @Parameter(description = "tenant/namespace", required = true)
 private java.util.List params;

[GitHub] [pulsar] sijie commented on issue #5592: [Issue 5589][pulsar-function-go] Fix a memory leak of pulsar-function-go library

[GitBox]

sijie commented on issue #5592: [Issue 5589][pulsar-function-go] Fix a memory 
leak of pulsar-function-go library
URL: https://github.com/apache/pulsar/pull/5592#issuecomment-561276427
 
 
   @wolfstudy thank you for your clarification! @Sunkwan-Kwon thank you for 
your contribution!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie merged pull request #5592: [Issue 5589][pulsar-function-go] Fix a memory leak of pulsar-function-go library

[GitBox]

sijie merged pull request #5592: [Issue 5589][pulsar-function-go] Fix a memory 
leak of pulsar-function-go library
URL: https://github.com/apache/pulsar/pull/5592
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie closed issue #5589: Memory leak of pulsar-function-go library

[GitBox]

sijie closed issue #5589: Memory leak of pulsar-function-go library
URL: https://github.com/apache/pulsar/issues/5589
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[pulsar] branch master updated: [Issue 5589][pulsar-function-go] Fix a memory leak of pulsar-function-go library (#5592)

[sijie]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 387e290  [Issue 5589][pulsar-function-go] Fix a memory leak of 
pulsar-function-go library (#5592)
387e290 is described below

commit 387e29079ef13a20c66ad77b97bede0a6d9814a2
Author: Sunkwan Kwon 
AuthorDate: Wed Dec 4 02:37:39 2019 +0900

[Issue 5589][pulsar-function-go] Fix a memory leak of pulsar-function-go 
library (#5592)

StrEntry variable should be cleared after finish addLogTopicHandler()
function regardless the log messages are appended to logger or not. If
it is not cleared, it causes memory leak because StrEntry has grown
indefinitely. Moreover, if the function set --log-topic, then the topic
could get accumulated huge messages that cause ProducerQueueIsFull
error.
---
 pulsar-function-go/pf/instance.go | 5 +
 1 file changed, 5 insertions(+)

diff --git a/pulsar-function-go/pf/instance.go 
b/pulsar-function-go/pf/instance.go
index ff25ebd..af28be2 100644
--- a/pulsar-function-go/pf/instance.go
+++ b/pulsar-function-go/pf/instance.go
@@ -289,6 +289,11 @@ func (gi *goInstance) setupLogHandler() error {
 }
 
 func (gi *goInstance) addLogTopicHandler() {
+   // Clear StrEntry regardless gi.context.logAppender is set or not
+   defer func() {
+   log.StrEntry = nil
+   }()
+
if gi.context.logAppender == nil {
log.Error("the logAppender is nil, if you want to use it, 
please specify `--log-topic` at startup.")
return

[GitHub] [pulsar] wmccarley opened a new issue #5792: Extend autoSkipNonRecoverableData to apply to missing Schemas in Bookkeeper

[GitBox]

wmccarley opened a new issue #5792: Extend autoSkipNonRecoverableData to apply 
to missing Schemas in Bookkeeper
URL: https://github.com/apache/pulsar/issues/5792
 
 
   **Background**
   Pulsar uses bookkeeper as the storage mechanism for data/cursors. In #1046 a 
flag was added by  @rdhabalia : _autoSkipNonRecoverableData_ that allows an 
admin to tell the broker to disregard missing ledgers and skip ahead. This 
helps in certain scenarios where the ledger is gone and it is unrecoverable (or 
not worth recovering.)
   
   This flag helps prevent users from getting stuck with 
NoSuchEntryException/NoLedgerException when the bookkeeper cluster has suffered 
data loss.
   
   **Is your feature request related to a problem? Please describe.**
   The _autoSkipNonRecoverableData_ flag does not solve the problem where the 
topic has a schema attached (stored in bookkeeper) and the ledger containing 
that schema goes missing. In this situation clients will receive 
NoSuchEntryException/NoLedgerException exceptions. Even if the admin unloads 
the topic the problem will still continue to occur until the admin deletes the 
schema
   
   There are two key problems:
   
   If the admin has never seen this issue before they will probably spend time 
checking stats-internal to see if the ledger is a cursor or if it contains 
entries. However the referenced ledger will not appear anywhere in 
stats-internal.
   
   The _autoSkipNonRecoverableData_ ostensibly exists to prevent users from 
getting stuck with NoSuchEntryException/NoLedgerException when ledgers go 
missing but it doesn't apply to missing schemas.
   
   **Describe the solution you'd like**
   
   Modify 
[BookkeeperSchemaStorage](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java)
 to check _autoSkipNonRecoverableData_ and behave accordingly.
   
   Add additional output to stats-internal to show schema information including 
the ledgers that are used for schema storage (if the default 
BookkeeperSchemaStorage implementation is being used.)
   
   **Describe alternatives you've considered**
   
   Rather than modify BookkeeperSchemaStorage the broker code could be modified 
to catch the missing ledger exception and hide it if this flag is set.
   
   A different boolean flag could also be used to control this behavior if 
there is a scenario where the admin wants to skip missing schemas but not 
missing data (or vice-versa)
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] cckellogg commented on a change in pull request #5790: [Issue:5687] Add prefix for new keys from Env

[GitBox]

cckellogg commented on a change in pull request #5790: [Issue:5687] Add prefix 
for new keys from Env
URL: https://github.com/apache/pulsar/pull/5790#discussion_r353257885
 
 

 ##
 File path: deployment/kubernetes/aws/broker.yaml
 ##
 @@ -25,15 +25,15 @@ metadata:
 data:
 # Tune for available memory. Increase the heap up to 24G to have
 # better GC behavior at high throughput
-PULSAR_MEM: "\" -Dio.netty.leakDetectionLevel=disabled 
-Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled 
-XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis 
-XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 
-XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError 
-XX:+PerfDisableSharedMem -Xms12g -Xmx12g -XX:MaxDirectMemorySize=14g 
-Dpulsar.root.logger=DEBUG,FILE \""
-PULSAR_GC: "\" -XX:+UseG1GC -XX:MaxGCPauseMillis=10\""
-zookeeperServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
-configurationStoreServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
-clusterName: us-east
-managedLedgerDefaultEnsembleSize: "2"
-managedLedgerDefaultWriteQuorum: "2"
-managedLedgerDefaultAckQuorum: "2"
-deduplicationEnabled: "false"
+PULSAR_PREFIX_PULSAR_MEM: "\" -Dio.netty.leakDetectionLevel=disabled 
-Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled 
-XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis 
-XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 
-XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError 
-XX:+PerfDisableSharedMem -Xms12g -Xmx12g -XX:MaxDirectMemorySize=14g 
-Dpulsar.root.logger=DEBUG,FILE \""
 
 Review comment:
   There is no need to put a prefix in front of PULSAR_MEM, PULSAR_GC, or 
PULSAR_EXTRA_OPTS. This script 
(https://github.com/apache/pulsar/blob/master/conf/pulsar_env.sh) will pick up 
the environment variables. Same for all the other places.
   
   Also, adding the PULSAR_PREFIX everywhere will add random variables to the 
configs. For example the broker is started with the following command
   ```
   bin/apply-config-from-env.py conf/broker.conf &&
   bin/apply-config-from-env.py conf/pulsar_env.sh &&
   bin/gen-yml-from-env.py conf/functions_worker.yml &&
   bin/pulsar broker
   ```
   Every PULSAR_PREFX_ key value will end up in the conf/pulsar_env.sh script. 
That might not matter now or break things but I don't think is the correct 
approach in general for the broker or proxy configs. In my opinion the override 
scripts should be completely removed from broker and proxy and function 
workers. There are ways to apply configs without doing search and replaces with 
scripts and i think this is cleaner and less error prone.
   
   For example, you can create a broker configmap with the configs you want to 
override:
   data:
 broker.conf: |-
   zookeeperServers=zk-0,zk-1,zk-2
   configurationStoreServers=zk-0,zk-1,zk-2
   clusterName=my-cluster
   
   This config can be mounted as a file in the container and then you can set 
this broker conf environment variable to where the file was mounted. For 
example:
   PULSAR_BROKER_CONF: /conf/broker.conf 
   
   Then you can just start the broker 
   ```
   ./bin/pulsar broker
   ```
   The broker will read the config you mounted through the configmap and merge 
it with the default values.
   
   If you want to add functions just add another config map with the function 
config and add the following flag to where the file was mounted.
   ```
   ./bin/pulsar broker --functions-worker-conf /conf/functions_worker.yml
   ```
   The same can be done for the proxy. I think this makes the deployments 
easier and removes the search and replace scripts from the brokers and proxy. 
The approach also allows you add any config in the future without having to 
update a script.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on issue #5465: [Pulsar Functions] User class constructor throws exception

[GitBox]

wolfstudy commented on issue #5465: [Pulsar Functions] User class constructor 
throws exception
URL: https://github.com/apache/pulsar/issues/5465#issuecomment-561210484
 
 
   > @wolfstudy Is it by design, or shall we add a note in the document?
   
   hello @Jennifer88huang the issue needs to be fixed, may be we need to 
refactor the logic.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Jennifer88huang commented on issue #5757: [Issue 5562][docs]Modify schema compatibility check doc

[GitBox]

Jennifer88huang commented on issue #5757: [Issue 5562][docs]Modify schema 
compatibility check doc
URL: https://github.com/apache/pulsar/pull/5757#issuecomment-561190035
 
 
   @congbobo184 @Anonymitaet When you fix an issue, please add the issue number 
and component in the title, for example, "[Issue 5562][docs] xxx", and add 
"Fixes #" in the description.
   So when the PR is merged, the corresponding issue will be closed 
automatically.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on issue #5669: Pulsar Broker restart fail owing to create functions/assignments topic fail

[GitBox]

wolfstudy commented on issue #5669: Pulsar Broker restart fail owing to create 
functions/assignments topic fail
URL: https://github.com/apache/pulsar/issues/5669#issuecomment-561174974
 
 
   Thanks @laxpio feedback.
   
   > the logs are cleaned.
   
   Can you provide the operation way when this error occurred or any 
information that can help locate this error? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on issue #5781: [Issue:4753] Add encryption docs to Functions develop

[GitBox]

wolfstudy commented on issue #5781: [Issue:4753] Add encryption docs to 
Functions develop
URL: https://github.com/apache/pulsar/pull/5781#issuecomment-561167088
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on a change in pull request #5767: Support batch authorization of partitioned topic

[GitBox]

tuteng commented on a change in pull request #5767: Support batch authorization 
of partitioned topic
URL: https://github.com/apache/pulsar/pull/5767#discussion_r353168621
 
 

 ##
 File path: 
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java
 ##
 @@ -312,4 +316,76 @@ public void testGetPartitionedTopicsList() throws 
KeeperException, InterruptedEx
 Assert.assertEquals(nonPersistentPartitionedTopics.size(), 1);
 
Assert.assertEquals(TopicName.get(nonPersistentPartitionedTopics.get(0)).getDomain().value(),
 TopicDomain.non_persistent.value());
 }
+
+@Test
+public void testGrantNonPartitionedTopic() {
+final String topicName = "non-partitioned-topic";
+persistentTopics.createNonPartitionedTopic(testTenant, testNamespace, 
topicName, true);
+String role = "role";
+Set expectActions = new HashSet<>();
+expectActions.add(AuthAction.produce);
+persistentTopics.grantPermissionsOnTopic(testTenant, testNamespace, 
topicName, role, expectActions);
+Map> permissions = 
persistentTopics.getPermissionsOnTopic(testTenant, testNamespace, topicName);
+Assert.assertEquals(permissions.get(role), expectActions);
+}
+
+@Test
+public void testGrantPartitionedTopic() {
+final String partitionedTopicName = "partitioned-topic";
+final int numPartitions = 5;
+persistentTopics.createPartitionedTopic(testTenant, testNamespace, 
partitionedTopicName, numPartitions);
+
+String role = "role";
+Set expectActions = new HashSet<>();
+expectActions.add(AuthAction.produce);
+persistentTopics.grantPermissionsOnTopic(testTenant, testNamespace, 
partitionedTopicName, role, expectActions);
+Map> permissions = 
persistentTopics.getPermissionsOnTopic(testTenant, testNamespace,
+partitionedTopicName);
+Assert.assertEquals(permissions.get(role), expectActions);
+TopicName topicName = TopicName.get(TopicDomain.persistent.value(), 
testTenant, testNamespace,
+partitionedTopicName);
+for (int i = 0; i < numPartitions; i++) {
+TopicName partition = topicName.getPartition(i);
+Map> partitionPermissions = 
persistentTopics.getPermissionsOnTopic(testTenant,
+testNamespace, partition.getEncodedLocalName());
 
 Review comment:
   I feel this call function `getEncodedLocalName` seems to be incorrect, the 
topic name here should have a suffix `-partition-0`. You can try the test in 
locally use the command:
   
   ```
   mvn test 
-Dtest=org.apache.pulsar.broker.admin.PersistentTopicsTest#testGrantPartitionedTopic
 -pl pulsar-broker
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on a change in pull request #5767: Support batch authorization of partitioned topic

[GitBox]

tuteng commented on a change in pull request #5767: Support batch authorization 
of partitioned topic
URL: https://github.com/apache/pulsar/pull/5767#discussion_r353166132
 
 

 ##
 File path: 
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java
 ##
 @@ -312,4 +316,82 @@ public void testGetPartitionedTopicsList() throws 
KeeperException, InterruptedEx
 Assert.assertEquals(nonPersistentPartitionedTopics.size(), 1);
 
Assert.assertEquals(TopicName.get(nonPersistentPartitionedTopics.get(0)).getDomain().value(),
 TopicDomain.non_persistent.value());
 }
+
+@Test
+public void testGrantNonPartitionedTopic() {
+final String topicName = "non-partitioned-topic";
+persistentTopics.createNonPartitionedTopic(testTenant, testNamespace, 
topicName, true);
+String role = "role";
+Set expectActions = new HashSet<>();
+expectActions.add(AuthAction.produce);
+persistentTopics.grantPermissionsOnTopic(testTenant, testNamespace, 
topicName, role, expectActions);
+Map> permissions = 
persistentTopics.getPermissionsOnTopic(testTenant, testNamespace, topicName);
+Assert.assertEquals(permissions.get(role), expectActions);
+}
+
+@Test
+public void testGrantPartitionedTopic() {
+final String partitionedTopicName = "partitioned-topic";
+final int numPartitions = 5;
+LocalZooKeeperCacheService mockLocalZooKeeperCacheService = 
mock(LocalZooKeeperCacheService.class);
+ZooKeeperChildrenCache mockZooKeeperChildrenCache = 
mock(ZooKeeperChildrenCache.class);
+
doReturn(mockLocalZooKeeperCacheService).when(pulsar).getLocalZkCacheService();
+
doReturn(mockZooKeeperChildrenCache).when(mockLocalZooKeeperCacheService).managedLedgerListCache();
 
 Review comment:
   I am not sure whether it is more appropriate to open a new API interface, 
because the current authorization and cancellation of permissions for both 
partitioned topic and non-partitioned topic are done under the same rest API. 
if opening a new rest API to get permissions causes confusion to users, what do 
you think? @sijie @jiazhai 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on a change in pull request #5767: Support batch authorization of partitioned topic

[GitBox]

tuteng commented on a change in pull request #5767: Support batch authorization 
of partitioned topic
URL: https://github.com/apache/pulsar/pull/5767#discussion_r353164610
 
 

 ##
 File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
 ##
 @@ -301,13 +295,29 @@ protected void internalGrantPermissionsOnTopic(String 
role, Set acti
 log.warn("[{}] Failed to grant permissions on topic {}: concurrent 
modification", clientAppId(),
 topicUri);
 throw new RestException(Status.CONFLICT, "Concurrent 
modification");
-}
-catch (Exception e) {
+} catch (Exception e) {
 log.error("[{}] Failed to grant permissions for topic {}", 
clientAppId(), topicUri, e);
 throw new RestException(e);
 }
 }
 
+protected void internalGrantPermissionsOnTopic(String role, 
Set actions) {
+// This operation should be reading from zookeeper and it should be 
allowed without having admin privileges
+validateAdminAccessForTenant(namespaceName.getTenant());
+validatePoliciesReadOnlyAccess();
+
+PartitionedTopicMetadata meta = getPartitionedTopicMetadata(topicName, 
true, false);
 
 Review comment:
   You are right. But one unit test never passed:
   
   You can test it locally:
   
   ```
   mvn test -Dtest=org.apache.pulsar.broker.admin.AdminTest#persistentTopics 
-pl pulsar-broker
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on issue #5635: [Issue 5612] Use Reactivestreams client for Mongo Sink

[GitBox]

tuteng commented on issue #5635: [Issue 5612] Use Reactivestreams client for 
Mongo Sink
URL: https://github.com/apache/pulsar/pull/5635#issuecomment-561156011
 
 
   run cpp tests
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] zymap commented on issue #5344: In Pulsar client, the effective value of NackRedeliveryDelay is about 1/3 of its specified value

[GitBox]

zymap commented on issue #5344: In Pulsar client, the effective value of 
NackRedeliveryDelay is about 1/3 of its specified value
URL: https://github.com/apache/pulsar/issues/5344#issuecomment-561148510
 
 
   Hi @Jesse-Zhang-Bose, I think exposing the `TimeInterval` might is not a 
good idea to address this.
   
   I will close this issue if you have not any problem.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] zymap opened a new pull request #5791: Add explanation about the v1 REST API

[GitBox]

zymap opened a new pull request #5791: Add explanation about the v1 REST API
URL: https://github.com/apache/pulsar/pull/5791
 
 
   ---
   
   Master Issue: #5683
   
   *Motivation*
   
   There is a sample tenant that will auto-create when
   a user starts a standalone service. Sometimes a user
   wants to do something in the sample tenant and but
   they will meet some errors when they using the pulsar
   admin to operate the sample tenant. We need to explain
   how to use the REST API with the old version of the
   namespace.
   
   *Modifications*
   
   - Add explanation in the 2.4.1 doc and master doc
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] alphashaw commented on issue #5770: Pulsar Broker Sometimes does not return CommandSendReceipt/CommandSendError to prducer send CommandSend

[GitBox]

alphashaw commented on issue #5770: Pulsar Broker Sometimes does not return 
CommandSendReceipt/CommandSendError to prducer send CommandSend
URL: https://github.com/apache/pulsar/issues/5770#issuecomment-561136400
 
 
   Yeah. I realized that after some few tests. I was using a queue to hold the 
request metadata to be poll when a responsive arrived. I realized, in some 
times, the head of the queue does not match the response. Thanks @jiazhai . 
   I have a working producer implementation here 
[Pulserl](https://github.com/alphashaw/pulserl). The README doc is very minimal 
right now. I'll give it more look over the weekend.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] zymap closed issue #4880: Cmake error when compiling pulsar-client-cpp

[GitBox]

zymap closed issue #4880: Cmake error when compiling pulsar-client-cpp
URL: https://github.com/apache/pulsar/issues/4880
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] zymap commented on issue #5714: Add more info in the error messages

[GitBox]

zymap commented on issue #5714: Add more info in the error messages
URL: https://github.com/apache/pulsar/pull/5714#issuecomment-561098054
 
 
   @sijie I add a new method `wrap` to wrap the exception to enriching the 
exception message info. 
   
   The wrap method will get the exception and wrap the old exception with the 
same type of the exception, so I can add more message info to the exception. I 
run the test that failed before, it can be passed on my laptop. 
   
   What do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy commented on a change in pull request #5790: [Issue:5687] Add prefix for new keys from Env

[GitBox]

wolfstudy commented on a change in pull request #5790: [Issue:5687] Add prefix 
for new keys from Env
URL: https://github.com/apache/pulsar/pull/5790#discussion_r353076109
 
 

 ##
 File path: deployment/kubernetes/generic/k8s-1-9-and-above/broker.yaml
 ##
 @@ -25,17 +25,17 @@ metadata:
 data:
 # Tune for available memory. Increase the heap up to 24G to have
 # better GC behavior at high throughput
-PULSAR_MEM: "\" -Xms64m -Xmx128m -XX:MaxDirectMemorySize=128m\""
-zookeeperServers: zookeeper
-configurationStoreServers: zookeeper
-clusterName: local
+PULSAR_PREFIX_PULSAR_MEM: "\" -Xms64m -Xmx128m 
-XX:MaxDirectMemorySize=128m\""
+PULSAR_PREFIX_zookeeperServers: zookeeper
+PULSAR_PREFIX_configurationStoreServers: zookeeper
+PULSAR_PREFIX_clusterName: local
 # change the managed ledger settings if you have more bookies
-managedLedgerDefaultEnsembleSize: "1"
-managedLedgerDefaultWriteQuorum: "1"
-managedLedgerDefaultAckQuorum: "1"
+PULSAR_PREFIX_managedLedgerDefaultEnsembleSize: "1"
+PULSAR_PREFIX_managedLedgerDefaultWriteQuorum: "1"
+PULSAR_PREFIX_managedLedgerDefaultAckQuorum: "1"
 # enable pulsar functions
-functionsWorkerEnabled: "true"
-PF_pulsarFunctionsCluster: local
 
 Review comment:
   Yes, will fix it


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jiazhai commented on a change in pull request #5790: [Issue:5687] Add prefix for new keys from Env

[GitBox]

jiazhai commented on a change in pull request #5790: [Issue:5687] Add prefix 
for new keys from Env
URL: https://github.com/apache/pulsar/pull/5790#discussion_r353070939
 
 

 ##
 File path: deployment/kubernetes/generic/k8s-1-9-and-above/broker.yaml
 ##
 @@ -25,17 +25,17 @@ metadata:
 data:
 # Tune for available memory. Increase the heap up to 24G to have
 # better GC behavior at high throughput
-PULSAR_MEM: "\" -Xms64m -Xmx128m -XX:MaxDirectMemorySize=128m\""
-zookeeperServers: zookeeper
-configurationStoreServers: zookeeper
-clusterName: local
+PULSAR_PREFIX_PULSAR_MEM: "\" -Xms64m -Xmx128m 
-XX:MaxDirectMemorySize=128m\""
+PULSAR_PREFIX_zookeeperServers: zookeeper
+PULSAR_PREFIX_configurationStoreServers: zookeeper
+PULSAR_PREFIX_clusterName: local
 # change the managed ledger settings if you have more bookies
-managedLedgerDefaultEnsembleSize: "1"
-managedLedgerDefaultWriteQuorum: "1"
-managedLedgerDefaultAckQuorum: "1"
+PULSAR_PREFIX_managedLedgerDefaultEnsembleSize: "1"
+PULSAR_PREFIX_managedLedgerDefaultWriteQuorum: "1"
+PULSAR_PREFIX_managedLedgerDefaultAckQuorum: "1"
 # enable pulsar functions
-functionsWorkerEnabled: "true"
-PF_pulsarFunctionsCluster: local
 
 Review comment:
   There was already a prefix for pulsar functions: 
   ```PF_ENV_PREFIX = 'PF_'```
   and `gen-yml-from-env.py` will handle it. we may not need to handle it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sky-big commented on issue #5534: Pulsar Operator On Kubernetes

[GitBox]

sky-big commented on issue #5534: Pulsar Operator On Kubernetes
URL: https://github.com/apache/pulsar/issues/5534#issuecomment-561082144
 
 
   Operator add pulsar manager component success.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jiazhai commented on issue #5668: bookie server's log error: Failed to restore rocksdb 000000000000000000/000000000000000001/000000000000000000

[GitBox]

jiazhai commented on issue #5668: bookie server's log error: Failed to restore 
rocksdb 00/01/00
URL: https://github.com/apache/pulsar/issues/5668#issuecomment-561079276
 
 
   Thanks @kaykillyou for your report. Since this is not a blocking issue. will 
put it in a low priority. 
   we will try to reproduce and fix the issue later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] wolfstudy opened a new pull request #5790: [Issue:5687] Add prefix for new keys from Env

[GitBox]

wolfstudy opened a new pull request #5790: [Issue:5687] Add prefix for new keys 
from Env
URL: https://github.com/apache/pulsar/pull/5790
 
 
   Signed-off-by: xiaolong.ran 
   
   Fixes #5687 
   
   ### Motivation
   
   When the user wants to add new keys for Env, adding fails if no prefix is 
added. 
   
   Currently, add new keys for Env use the script of 
[apply-config-from-env.py](https://github.com/apache/pulsar/commits/master/docker/pulsar/scripts/apply-config-from-env.py),
 to ensure that the env set by the user can take effect, add the 
prefix(**PULSAR_PREFIX_**) for all keys.
   
   ### Modifications
   
   - Add prefix for new keys from Env
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] lspgn edited a comment on issue #5695: Go: Segfault on messageId serialize after getting it from producer

[GitBox]

lspgn edited a comment on issue #5695: Go: Segfault on messageId serialize 
after getting it from producer
URL: https://github.com/apache/pulsar/issues/5695#issuecomment-561059864
 
 
   I created a branch with more debug:
   
https://github.com/lspgn/pulsar/commit/7d1c0f4c2074f2240b888306822d81092042a490
   
https://github.com/lspgn/pulsar/commit/1373b1b884a6b6822568ca4e931445469683f72f
   
   The following C prints more information about the messageId structure:
   ```
   static int myprint(pulsar_message_id_t* messageId) {
printf("  messageId: %x\n", messageId);
char *a = messageId;
printf("  strlen: %d\n", strlen(a));
printf("  ");
for(int i = 0; i < strlen(a);i++) {
printf("0x%x ", a[i]);
}
printf("\n");
return strlen
   }
   ```
   
   The output of this is:
   ```
   
   getMessageId
   messageID String &{}
 messageId: 7811e60
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6f002a0
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6f002a0
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 7a086a0
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6e014c0
 strlen: 3
 0xff9f 0x2 0x6e
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6f02a80
 strlen: 4
 0xffb8 0x2a 0xfff0 0x6
   The message Id value is: [(75556568,0,2433,37)]
   getMessageId
   messageID String &{}
 messageId: 6e00c10
 strlen: 4
 0xffc8 0x2b 0xffe0 0x6
   The message Id value is: [(75556568,0,2433,37)]
   getMessageId
   messageID String &{}
 messageId: 7812940
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6e01f40
 strlen: 3
 0xffb4 0x2 0x6e
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 7a086a0
 strlen: 0
   
   ```
   It seems to only work when the length of the ID is 4 bytes. Otherwise it 
fails to convert to a proper ID.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] lspgn edited a comment on issue #5695: Go: Segfault on messageId serialize after getting it from producer

[GitBox]

lspgn edited a comment on issue #5695: Go: Segfault on messageId serialize 
after getting it from producer
URL: https://github.com/apache/pulsar/issues/5695#issuecomment-561059864
 
 
   I created a branch with more debug:
   
https://github.com/lspgn/pulsar/commit/7d1c0f4c2074f2240b888306822d81092042a490
   
https://github.com/lspgn/pulsar/commit/1373b1b884a6b6822568ca4e931445469683f72f
   
   The following C prints more information about the messageId structure:
   ```
   static int myprint(pulsar_message_id_t* messageId) {
printf("  messageId: %x\n", messageId);
char *a = messageId;
printf("  strlen: %d\n", strlen(a));
printf("  ");
for(int i = 0; i < strlen(a);i++) {
printf("0x%x ", a[i]);
}
printf("\n");
return strlen
   }
   ```
   
   ```
   func (m *messageID) String() string {
fmt.Printf("messageID String %v\n", m.ptr)
strl := int(C.myprint(m.ptr))
   
if strl == 4 {
str := C.pulsar_message_id_str(m.ptr)
defer C.free(unsafe.Pointer(str))
return C.GoString(str)  
}

return "none"
   }
   ```
   
   The output of this is:
   ```
   
   getMessageId
   messageID String &{}
 messageId: 7811e60
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6f002a0
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6f002a0
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 7a086a0
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6e014c0
 strlen: 3
 0xff9f 0x2 0x6e
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6f02a80
 strlen: 4
 0xffb8 0x2a 0xfff0 0x6
   The message Id value is: [(75556568,0,2433,37)]
   getMessageId
   messageID String &{}
 messageId: 6e00c10
 strlen: 4
 0xffc8 0x2b 0xffe0 0x6
   The message Id value is: [(75556568,0,2433,37)]
   getMessageId
   messageID String &{}
 messageId: 7812940
 strlen: 0
   
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 6e01f40
 strlen: 3
 0xffb4 0x2 0x6e
   The message Id value is: [none]
   getMessageId
   messageID String &{}
 messageId: 7a086a0
 strlen: 0
   
   ```
   It seems to only work when the length of the ID is 4 bytes. Otherwise it 
fails to convert to a proper ID.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] lspgn commented on issue #5695: Go: Segfault on messageId serialize after getting it from producer

[GitBox]

lspgn commented on issue #5695: Go: Segfault on messageId serialize after 
getting it from producer
URL: https://github.com/apache/pulsar/issues/5695#issuecomment-561059864
 
 
   I created a branch with more debug:
   
https://github.com/lspgn/pulsar/commit/7d1c0f4c2074f2240b888306822d81092042a490
   
   The following C prints more information about the messageId structure:
   ```
   static void myprint(pulsar_message_id_t* messageId) {
printf("  messageId: %x\n", messageId);
char *a = messageId;
printf("  strlen: %d\n", strlen(a));
printf("  ");
for(int i = 0; i < strlen(a);i++) {
printf("0x%x ", a[i]);
}
printf("\n");
   }
   ```
   
   The output of this is:
   ```
   
   getMessageId
   messageID String &{}
 messageId: a401c60
 strlen: 0
   
   The message Id value is: [6e6f6e65]
   getMessageId
   messageID String &{}
 messageId: a401c60
 strlen: 0
   
   The message Id value is: [6e6f6e65]
   getMessageId
   messageID String &{}
 messageId: a401c60
 strlen: 0
   
   The message Id value is: [6e6f6e65]
   getMessageId
   messageID String &{}
 messageId: a300fa0
 strlen: 0
   
   The message Id value is: [6e6f6e65]
   getMessageId
   messageID String &{}
 messageId: 6e032f0
 strlen: 3
 0xff93 0x2 0x6e
   The message Id value is: [6e6f6e65]
   getMessageId
   messageID String &{}
 messageId: 6f16380
 strlen: 4
 0xfff8 0x63 0xfff1 0x6
   The message Id value is: [6e6f6e65]
   getMessageId
   messageID String &{}
 messageId: 6f165c0
 strlen: 0
   
   The message Id value is: [6e6f6e65]
   getMessageId
   messageID String &{}
 messageId: 6e035a0
 strlen: 4
 0xffc8 0x76 0xffe0 0x6
   The message Id value is: [6e6f6e65]
   getMessageId
   messageID String &{}
 messageId: a402770
 strlen: 4
 0xffe8 0x2c 0x40 0xa
   The message Id value is: [6e6f6e65]
   getMessageId
   messageID String &{}
 messageId: 6f16380
 strlen: 4
 0x48 0x67 0xfff1 0x6
   The message Id value is: [6e6f6e65]
   ```
   
   Quite a few messages have a length of zero?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Jennifer88huang closed issue #5222: 2.4.1 pulsar-admin should have 8 schema compatibility check strategies

[GitBox]

Jennifer88huang closed issue #5222: 2.4.1 pulsar-admin should have 8 schema 
compatibility check strategies
URL: https://github.com/apache/pulsar/issues/5222
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet commented on issue #4704: Add documentation for HDFS offloader

[GitBox]

Anonymitaet commented on issue #4704: Add documentation for HDFS offloader
URL: https://github.com/apache/pulsar/issues/4704#issuecomment-561054680
 
 
   Fixed in https://github.com/apache/pulsar/pull/5762.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services