[GitHub] [pulsar] candlerb commented on issue #5521: Python exception message not shown when python function fails to start
candlerb commented on issue #5521: Python exception message not shown when python function fails to start URL: https://github.com/apache/pulsar/issues/5521#issuecomment-548376464 Argh, I found the logs under `apache-pulsar-2.4.1/logs/functions/public/default/womble/`, not on pulsar output. Sorry for the noise! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb closed issue #5521: Python exception message not shown when python function fails to start
candlerb closed issue #5521: Python exception message not shown when python function fails to start URL: https://github.com/apache/pulsar/issues/5521 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se commented on issue #5518: python functions cannot find python3 interpreter
aahmed-se commented on issue #5518: python functions cannot find python3 interpreter URL: https://github.com/apache/pulsar/issues/5518#issuecomment-548444816 the question would be what is the default option if any. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se commented on issue #5516: COMPILATION ERROR:pulsar-kafka-compat-client-test
aahmed-se commented on issue #5516: COMPILATION ERROR:pulsar-kafka-compat-client-test URL: https://github.com/apache/pulsar/issues/5516#issuecomment-548449764 did you build the dependent modules ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] apankha opened a new issue #5523: Similar to issue #2037 - Initialization of cluster meta data fails with KeeperErrorCode = NodeExists
apankha opened a new issue #5523: Similar to issue #2037 - Initialization of cluster meta data fails with KeeperErrorCode = NodeExists URL: https://github.com/apache/pulsar/issues/5523 Very similar issue to: https://github.com/apache/pulsar/issues/2037 Initialization of cluster meta data fails while setting up a cluster pulsar instance following the steps in https://pulsar.apache.org/docs/en/deploy-bare-metal/ **Actual behaviour:** Exception in thread "main" org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /namespace at org.apache.zookeeper.KeeperException.create(KeeperException.java:122) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) at org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:175) **Expected behavior** We have managed to set up a similar cluster with RHEL7 VMs with similar configuration without any issues. **System Configuration** Pulsar version 2.4.1 Java version zulu8.36.0.1-ca-jdk8.0.202-linux_x64 Any suggestions to help us resolve the problem? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb opened a new pull request #5524: Doc: add missing functions from Python context object
candlerb opened a new pull request #5524: Doc: add missing functions from Python context object URL: https://github.com/apache/pulsar/pull/5524 ### Motivation Improve documentation ### Modifications * At [functions-api/#python-context-object](https://pulsar.apache.org/docs/en/next/functions-api/#python-context-object), list the python methods which were missing * Add Python tab to [functions-develop/#context](https://pulsar.apache.org/docs/en/next/functions-develop/#context) NOTE: functions-api is not linked from the nav bar, that probably should be fixed too ### Verifying this change This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: N/A ### Documentation Doc update only This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5515: [Transaction Coordinator] Bootstrap pulsar system namespace and create TC assign topic.
codelipenghui commented on issue #5515: [Transaction Coordinator] Bootstrap pulsar system namespace and create TC assign topic. URL: https://github.com/apache/pulsar/pull/5515#issuecomment-548422184 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5504: [Transaction coordinator]Add transaction metadata store service
codelipenghui commented on issue #5504: [Transaction coordinator]Add transaction metadata store service URL: https://github.com/apache/pulsar/pull/5504#issuecomment-548421702 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb closed issue #5522: localrun fails to load python functions from correct file
candlerb closed issue #5522: localrun fails to load python functions from correct file URL: https://github.com/apache/pulsar/issues/5522 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb commented on issue #5522: localrun fails to load python functions from correct file
candlerb commented on issue #5522: localrun fails to load python functions from correct file URL: https://github.com/apache/pulsar/issues/5522#issuecomment-548383804 Argh, this is user error (although IMO documentation could be much better). Required usage is: ``` --py /home/ubuntu/func1.py --classname func1.FirstFunction ``` Note required duplication of "func1". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb commented on issue #5518: python functions cannot find python3 interpreter
candlerb commented on issue #5518: python functions cannot find python3 interpreter URL: https://github.com/apache/pulsar/issues/5518#issuecomment-548458634 > the question would be what is the default option if any. I would be fine with the default remaining at "python" for backwards-compatibility reasons, as long as there's a way to override it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb opened a new issue #5522: localrun fails to load python functions from correct file
candlerb opened a new issue #5522: localrun fails to load python functions from
correct file
URL: https://github.com/apache/pulsar/issues/5522
**Describe the bug**
If you try to run a function like this:
```
apache-pulsar-2.4.1/bin/pulsar-admin functions localrun ... --py
/home/ubuntu/func1.py
```
Then the function fails to start. Strace output shows that it's trying to
read python code from `/home/ubuntu` instead of `/home/ubuntu/func1.py`
**To Reproduce**
As per #5521
I have `func1.py` in the current directory:
```
$ pwd
/home/ubuntu
$ cat func1.py
from pulsar import Function
class FirstFunction(Function):
def process(self, item, context):
log = context.get_logger()
log.info("Got %r with properties %r" % (item,
context.get_message_properties()))
```
What I originally wanted to do was run it like this:
```
$ apache-pulsar-2.4.1/bin/pulsar-admin functions localrun --name womble2
--inputs my-topic --py func1.py --classname FirstFunction --log-topic pulsar-log
The specified python file does not exist
...
```
That's because pulsar-admin changes the working directory, as almost the
first thing it does. So instead I gave an absolute path to the function:
```
$ apache-pulsar-2.4.1/bin/pulsar-admin functions localrun --name womble2
--inputs my-topic --py /home/ubuntu/func1.py --classname FirstFunction
--log-topic pulsar-log
```
Pulsar goes into a loop, attempting to start every 30 seconds, but not
indicating why the python process fails. (Full session logs in #5521, that
ticket is about the error output beig discarded)
So I run under strace:
```
$ strace -f -s128 apache-pulsar-2.4.1/bin/pulsar-admin functions localrun
--name womble2 --inputs my-topic --py /home/ubuntu/func1.py --classname
FirstFunction --log-topic pulsar-log 2>/tmp/strace.out
...
13:14:07.229 [main] INFO org.apache.pulsar.functions.LocalRunner -
RuntimeSpawner quit because of
^C
```
And buried in the strace is the error message:
```
$ grep "Failed to import class" /tmp/strace.out
[pid 17173] write(3, "[2019-10-31 12:48:33 +] [INFO] util.py: Failed to
import class FirstFunction from path /home/ubuntu\n", 104) = 104
```
It should be trying to read from `/home/ubuntu/func1.py`
So then I tried moving `func1.py` into the `apache-pulsar-2.4.1` directory,
and running like this:
```
$ strace -s128 -f apache-pulsar-2.4.1/bin/pulsar-admin functions localrun
--name womble2 --inputs my-topic --py func1.py --classname FirstFunction
--log-topic pulsar-log 2>/tmp/strace.out
```
Now the file it is trying to read from is an empty filename!
```
$ grep "Failed to import class" /tmp/strace.out
[pid 18278] write(3, "[2019-10-31 13:14:07 +] [INFO] util.py: Failed to
import class FirstFunction from path \n", 92) = 92
```
**Expected behavior**
Function to be loaded from the file I specified
**Screenshots**
See above
**Desktop (please complete the following information):**
- Ubuntu 18.04
**Additional context**
N/A
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] PierreZ commented on issue #5495: Add a reverse mapping from role to namespaces
PierreZ commented on issue #5495: Add a reverse mapping from role to namespaces URL: https://github.com/apache/pulsar/issues/5495#issuecomment-548377871 > > True, I was thinking about implementing your first proposal. Something that is looping through znodes, and check if auth_policies.namespace_auth.containsKey(role) for the default implementation. > > That might be expensive if called frequently from clients in any case with a non-trivial number of namespaces. > > > I must admit that I saw only one implementation within the codebase, so I might underestimate this particular point. > > Yes, there's just one implementation in the codebase, the default where ACLs are stored with the namespace policies on ZK, though there are others using different authorization options, including : > > * Use AuthZ information stored in the AuthN credentials (eg: JWT token, or TLS certificate, also containing which resources this token has access to. > > * Third party ACLs provider (eg: Okta, Auth0, Vault, etc.. ) Thanks a lot @merlimat for all your insights How this issue should be adressed in your opinion? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb commented on issue #5520: Missing documentation for window functions
candlerb commented on issue #5520: Missing documentation for window functions URL: https://github.com/apache/pulsar/issues/5520#issuecomment-548390399 I got part of the answer from pulsar: ``` $ apache-pulsar-2.4.1/bin/pulsar-admin functions update --name womble --window-length-count 5 --sliding-interval-count 3 There is currently no support windowing in python Reason: There is currently no support windowing in python ``` But I'd still like to see documentation for what windowing is supposed to do :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb edited a comment on issue #5518: python functions cannot find python3 interpreter
candlerb edited a comment on issue #5518: python functions cannot find python3
interpreter
URL: https://github.com/apache/pulsar/issues/5518#issuecomment-548351376
Part of the answer comes from pulsar's debug output: it is running `python
.../python_instance_main.py ...args...`
```
12:25:45.004 [function-timer-thread-94-1] INFO
org.apache.pulsar.functions.runtime.ProcessRuntime - ProcessBuilder starting
the process with args
python
/home/ubuntu/apache-pulsar-2.4.1/instances/python-instance/python_instance_main.py
--py /tmp/pulsar_functions/public/default/womble/0/func1.py
--logging_directory /home/ubuntu/apache-pulsar-2.4.1/logs/functions
--logging_file womble
--logging_config_file
/home/ubuntu/apache-pulsar-2.4.1/conf/functions-logging/logging_config.ini
--instance_id 0 --function_id 67fb9828-56ab-49ff-bbbc-56db9510d26c
--function_version 4f01d696-28da-40b7-b9f5-2283fbe9bd7c
--function_details
'{"tenant":"public","namespace":"default","name":"womble","className":"FirstFunction","logTopic":"pulsar-log","runtime":"PYTHON","autoAck":true,"parallelism":1,"source":{"inputSpecs":{"my-topic":{}},"cleanupSubscription":true},"sink":{},"resources":{"cpu":1.0,"ram":"1073741824","disk":"10737418240"},"componentType":"FUNCTION"}'
--pulsar_serviceurl pulsar://ldex-pulsar.int.example.net:6650
--max_buffered_tuples 1024
--port 38423 --metrics_port 36591 --state_storage_serviceurl
bk://127.0.0.1:4181
--expected_healthcheck_interval 30 --secrets_provider
secretsprovider.ClearTextSecretsProvider
--cluster_name standalone
```
So a configuration option to change "python" to "python3" here would be
great, and solve the problem.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] aahmed-se commented on issue #5514: use pulsar get error in mac os
aahmed-se commented on issue #5514: use pulsar get error in mac os URL: https://github.com/apache/pulsar/issues/5514#issuecomment-548450966 @nathan-zhu you are naming your file pulsar.py it's confusing python This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb commented on issue #5526: Python API: add get_input_topics()
candlerb commented on issue #5526: Python API: add get_input_topics()
URL: https://github.com/apache/pulsar/pull/5526#issuecomment-548519972
Failed test is cpp, appears unrelated to this change.
```
[34/160] BasicEndToEndTest.testResendViaSendCallback (4234 ms)
[0;33mNote: Google Test filter = BasicEndToEndTest.testResendViaSendCallback
[m[0;32m[==] [mRunning 1 test from 1 test case.
[0;32m[--] [mGlobal test environment set-up.
[0;32m[--] [m1 test from BasicEndToEndTest
[0;32m[ RUN ] [mBasicEndToEndTest.testResendViaSendCallback
2019-10-31 18:43:54.603 INFO ConnectionPool:85 | Created connection for
pulsar://localhost:6650
2019-10-31 18:43:54.611 INFO ClientConnection:324 | [127.0.0.1:37190 ->
127.0.0.1:6650] Connected to broker
2019-10-31 18:43:54.794 INFO BatchMessageContainer:43 | { BatchContainer
[size = 0] [batchSizeInBytes_ = 0] [maxAllowedMessageBatchSizeInBytes_ =
131072] [maxAllowedNumMessagesInBatch_ = 1000] [topicName =
persistent://public/default/testResendViaListener] [producerName_ = ]
[batchSizeInBytes_ = 0] [numberOfBatchesSent = 0] [averageBatchSize = 0]}
BatchMessageContainer constructed
2019-10-31 18:43:54.795 INFO HandlerBase:53 |
[persistent://public/default/testResendViaListener, ] Getting connection from
pool
2019-10-31 18:43:55.294 INFO ProducerImpl:151 |
[persistent://public/default/testResendViaListener, ] Created producer on
broker [127.0.0.1:37190 -> 127.0.0.1:6650]
[34/160] BasicEndToEndTest.testResendViaSendCallback returned/aborted with
exit code -11 (4234 ms)
...
FAILED TESTS (1/160):
4234 ms: ./main BasicEndToEndTest.testResendViaSendCallback
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat merged pull request #5524: Doc: add missing functions from Python context object
merlimat merged pull request #5524: Doc: add missing functions from Python context object URL: https://github.com/apache/pulsar/pull/5524 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (d9ebec5 -> bcce54d)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from d9ebec5 Python API: Add test case for create_reader(is_read_compacted) (#5488) add bcce54d Doc: add missing functions from Python context object (#5524) No new revisions were added by this update. Summary of changes: site2/docs/functions-api.md | 10 +++ site2/docs/functions-develop.md | 66 + 2 files changed, 76 insertions(+)
[GitHub] [pulsar] merlimat merged pull request #5423: [Test] Migrate deprecated Matchers to ArgumentMatchers
merlimat merged pull request #5423: [Test] Migrate deprecated Matchers to ArgumentMatchers URL: https://github.com/apache/pulsar/pull/5423 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5491: Fix message deduplicate issue while using external sequence id with batch produce
merlimat commented on a change in pull request #5491: Fix message deduplicate
issue while using external sequence id with batch produce
URL: https://github.com/apache/pulsar/pull/5491#discussion_r341263375
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Topic.java
##
@@ -76,6 +76,14 @@ default long getOriginalSequenceId() {
}
void completed(Exception e, long ledgerId, long entryId);
+
+default long getLowestSequenceId() {
Review comment:
> It is in the inner class named PublishContext of Topic
Which already contains a sequence id field
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5491: Fix message deduplicate issue while using external sequence id with batch produce
merlimat commented on a change in pull request #5491: Fix message deduplicate
issue while using external sequence id with batch produce
URL: https://github.com/apache/pulsar/pull/5491#discussion_r341263026
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java
##
@@ -257,6 +285,9 @@ public ServerCnx getCnx() {
private String originalProducerName;
private long originalSequenceId;
+private long lowestSequenceId;
Review comment:
Ok, though as in other comment, there's already a `sequenceId` field. Is
that now ignored? If yes, then it should be removed.
Also, as you can see, there's an `originalSequenceId` field. This is used in
the context of geo-replication and it would have to be accounted for as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5491: Fix message deduplicate issue while using external sequence id with batch produce
merlimat commented on a change in pull request #5491: Fix message deduplicate
issue while using external sequence id with batch produce
URL: https://github.com/apache/pulsar/pull/5491#discussion_r341265357
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -367,6 +371,11 @@ public void sendAsync(Message message, SendCallback
callback) {
msgMetadataBuilder.setSequenceId(sequenceId);
} else {
sequenceId = msgMetadataBuilder.getSequenceId();
+if (sequenceId <= lastSequenceIdPushed) {
+callback.sendComplete(new PulsarClientException
+.InvalidMessageException("Message is
definitely a duplicate"));
Review comment:
This is not correct on 2 levels:
1. If `sequenceId <= lastSequenceIdPushed` we don't know yet whether the
message is already dup, because the previous attempt might still fail. This has
to be disambiguated by the broker which has visibility at the storage level.
2. As mentioned before, we cannot throw error when there's a duplicate,
rather we need to return "ok" to the application.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5423: [Test] Migrate deprecated Matchers to ArgumentMatchers
sijie commented on issue #5423: [Test] Migrate deprecated Matchers to ArgumentMatchers URL: https://github.com/apache/pulsar/pull/5423#issuecomment-548501913 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on a change in pull request #5463: Function runtime pluggable
jerrypeng commented on a change in pull request #5463: Function runtime
pluggable
URL: https://github.com/apache/pulsar/pull/5463#discussion_r341292908
##
File path:
pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java
##
@@ -328,162 +332,20 @@ public boolean getTlsEnabled() {
}
-@Data
Review comment:
@sijie I didn't remove them, I moved them down to here:
https://github.com/apache/pulsar/pull/5463/files#diff-a26c05f28ea3e67193e9ad83c63f687fR435
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5504: [Transaction coordinator]Add transaction metadata store service
sijie commented on issue #5504: [Transaction coordinator]Add transaction metadata store service URL: https://github.com/apache/pulsar/pull/5504#issuecomment-548493110 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie closed pull request #5431: [Issue 5430][pulsar-flink] Flink connector sink should implement consumeDataStream for 1.9.0
sijie closed pull request #5431: [Issue 5430][pulsar-flink] Flink connector sink should implement consumeDataStream for 1.9.0 URL: https://github.com/apache/pulsar/pull/5431 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on a change in pull request #5463: Function runtime pluggable
jerrypeng commented on a change in pull request #5463: Function runtime
pluggable
URL: https://github.com/apache/pulsar/pull/5463#discussion_r341292908
##
File path:
pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java
##
@@ -328,162 +332,20 @@ public boolean getTlsEnabled() {
}
-@Data
Review comment:
@sijie I didn't remove them, I moved them done to here:
https://github.com/apache/pulsar/pull/5463/files#diff-a26c05f28ea3e67193e9ad83c63f687fR435
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5431: [Issue 5430][pulsar-flink] Flink connector sink should implement consumeDataStream for 1.9.0
sijie commented on issue #5431: [Issue 5430][pulsar-flink] Flink connector sink should implement consumeDataStream for 1.9.0 URL: https://github.com/apache/pulsar/pull/5431#issuecomment-548501807 As discussed in #5430, close this issue in favor of the integration in Flink community. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #5463: Function runtime pluggable
jerrypeng commented on issue #5463: Function runtime pluggable URL: https://github.com/apache/pulsar/pull/5463#issuecomment-548524006 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #5358: PIP-45: Switch ManagedLedger to use MetadataStore interface
merlimat commented on issue #5358: PIP-45: Switch ManagedLedger to use MetadataStore interface URL: https://github.com/apache/pulsar/pull/5358#issuecomment-548480515 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5515: [Transaction Coordinator] Bootstrap pulsar system namespace and create TC assign topic.
sijie commented on a change in pull request #5515: [Transaction Coordinator]
Bootstrap pulsar system namespace and create TC assign topic.
URL: https://github.com/apache/pulsar/pull/5515#discussion_r341284909
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
##
@@ -472,6 +472,11 @@ public Boolean get() {
// Register heartbeat and bootstrap namespaces.
this.nsService.registerBootstrapNamespaces();
+// Register pulsar system namespaces
+if (config.isTransactionCoordinatorEnabled()) {
+
this.nsService.registerNamespace(NamespaceName.SYSTEM_NAMESPACE.toString(),
false);
Review comment:
We might need to document this. If a user already has a Pulsar cluster (e.g.
2.4.0), the user wants to upgrade to 2.5.0 to enable transaction. He/she has to
create a `pulsar/system` namespace and the coordinator topic. So it might be
making sense to add a command tool (similar as PulsarClusterMetadataSetup) to
upgrade the cluster to enable transaction.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5515: [Transaction Coordinator] Bootstrap pulsar system namespace and create TC assign topic.
sijie commented on a change in pull request #5515: [Transaction Coordinator]
Bootstrap pulsar system namespace and create TC assign topic.
URL: https://github.com/apache/pulsar/pull/5515#discussion_r341283033
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
##
@@ -135,6 +144,10 @@ public static void main(String[] args) throws Exception {
arguments.configurationStore = arguments.globalZookeeper;
}
+if (arguments.numTransactionCoordinators <= 0) {
+System.err.println("Number of transaction coordinators must
greater than 0");
Review comment:
don't you need to exit here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5512: [Issue 5475][docs] Update message deduplication
sijie commented on issue #5512: [Issue 5475][docs] Update message deduplication URL: https://github.com/apache/pulsar/pull/5512#issuecomment-548500449 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5502: Fix list non-persistent topics shows the persistent topics
sijie commented on issue #5502: Fix list non-persistent topics shows the persistent topics URL: https://github.com/apache/pulsar/pull/5502#issuecomment-548500769 @codelipenghui PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on a change in pull request #5463: Function runtime pluggable
jerrypeng commented on a change in pull request #5463: Function runtime
pluggable
URL: https://github.com/apache/pulsar/pull/5463#discussion_r341294347
##
File path:
pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java
##
@@ -328,162 +332,20 @@ public boolean getTlsEnabled() {
}
-@Data
Review comment:
There is also a test to make sure BC is preserved:
https://github.com/apache/pulsar/pull/5463/files#diff-7eb172c16a86f29eeec8f23f17993cdcR829
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5463: Function runtime pluggable
sijie commented on a change in pull request #5463: Function runtime pluggable
URL: https://github.com/apache/pulsar/pull/5463#discussion_r341286827
##
File path:
pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java
##
@@ -328,162 +332,20 @@ public boolean getTlsEnabled() {
}
-@Data
Review comment:
Removing configuration is usually treated as a breaking change. Can we keep
the old configuration settings for one release but deprecate them? Then after a
few more releases, we can remove the settings. Otherwise, it is painful for
people to upgrade a cluster.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5517: expose new message with different schema
sijie commented on a change in pull request #5517: expose new message with different schema URL: https://github.com/apache/pulsar/pull/5517#discussion_r341289310 ## File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java ## @@ -418,7 +418,19 @@ *the list of interceptors to intercept the producer created by this builder. * @return the producer builder instance */ -ProducerBuilder intercept(ProducerInterceptor ... interceptors); +@Deprecated +ProducerBuilder intercept(ProducerInterceptor ... interceptors); Review comment: Can you clarify the change here? Will that break binary compatibility? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb opened a new issue #5525: python API missing get_input_topics
candlerb opened a new issue #5525: python API missing get_input_topics URL: https://github.com/apache/pulsar/issues/5525 **Is your feature request related to a problem? Please describe.** Java and Go APIs have getInputTopics / GetInputTopics in the context object. This is missing from the Python API - although it has [get_output_topic](https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/python/contextimpl.py#L138) **Describe the solution you'd like** Add `get_input_topics()` **Describe alternatives you've considered** N/A **Additional context** N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb opened a new pull request #5526: Python API: add get_input_topics()
candlerb opened a new pull request #5526: Python API: add get_input_topics() URL: https://github.com/apache/pulsar/pull/5526 Fixes #5525 ### Motivation Feature parity with Java and Go APIs ### Modifications Add `get_input_topics()` to Python Context/ContextImpl ### Verifying this change * This change is a trivial rework / code cleanup without any test coverage. (There are no existing test cases for other ContextImpl methods like `get_output_topic`. But I have tested this locally within a pulsar function) ### Does this pull request potentially affect one of the following parts: - The public API: (python) adds new method ### Documentation - These methods missing from documentation anyway. PR #5524 adds `get_output_topic`; `get_input_topics` can be added once that has been merged (or if this one is merged first, then I'll update #5524) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (bcce54d -> f01ff8a)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from bcce54d Doc: add missing functions from Python context object (#5524) add f01ff8a [Issue 5475][docs] Update message deduplication (#5512) No new revisions were added by this update. Summary of changes: site2/docs/cookbooks-deduplication.md | 63 +-- 1 file changed, 31 insertions(+), 32 deletions(-)
[pulsar] branch master updated (f01ff8a -> 3d5bc0a)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from f01ff8a [Issue 5475][docs] Update message deduplication (#5512) add 3d5bc0a [Test] Migrate deprecated Matchers to ArgumentMatchers (#5423) No new revisions were added by this update. Summary of changes: .../pulsar/broker/service/ServerCnxTest.java | 2 +- .../buffer/PersistentTransactionBufferTest.java| 27 +++--- .../clients/producer/PulsarKafkaProducerTest.java | 9 3 files changed, 18 insertions(+), 20 deletions(-)
[GitHub] [pulsar] merlimat merged pull request #5512: [Issue 5475][docs] Update message deduplication
merlimat merged pull request #5512: [Issue 5475][docs] Update message deduplication URL: https://github.com/apache/pulsar/pull/5512 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat closed issue #5475: Doc display issue on message deduplication page
merlimat closed issue #5475: Doc display issue on message deduplication page URL: https://github.com/apache/pulsar/issues/5475 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] addisonj commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.6
addisonj commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.6 URL: https://github.com/apache/pulsar/pull/5043#issuecomment-548659715 rerun java8 tests rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Jennifer88huang commented on issue #5519: Zookeeper issue
Jennifer88huang commented on issue #5519: Zookeeper issue URL: https://github.com/apache/pulsar/issues/5519#issuecomment-548669299 @jiazhai could you please help look into the issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Jennifer88huang commented on issue #5520: Missing documentation for window functions
Jennifer88huang commented on issue #5520: Missing documentation for window functions URL: https://github.com/apache/pulsar/issues/5520#issuecomment-548669020 @candlerb Thank you very much for raising the issue. I'll look into the issue, and provide related docs soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] yittg commented on a change in pull request #5517: expose new message with different schema
yittg commented on a change in pull request #5517: expose new message with different schema URL: https://github.com/apache/pulsar/pull/5517#discussion_r341455971 ## File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java ## @@ -418,7 +418,19 @@ *the list of interceptors to intercept the producer created by this builder. * @return the producer builder instance */ -ProducerBuilder intercept(ProducerInterceptor ... interceptors); +@Deprecated +ProducerBuilder intercept(ProducerInterceptor ... interceptors); Review comment: It's not required to change this signature, i'll fix it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase
wolfstudy commented on issue #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase URL: https://github.com/apache/pulsar/pull/5527#issuecomment-548653356 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] addisonj commented on issue #5400: [functions] Allow functions to pass runtime specific options
addisonj commented on issue #5400: [functions] Allow functions to pass runtime specific options URL: https://github.com/apache/pulsar/pull/5400#issuecomment-548664048 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (79f4710 -> dd58baf)
This is an automated email from the ASF dual-hosted git repository. rxl pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 79f4710 Fix list non-persistent topics shows the persistent topics (#5502) add dd58baf fix: Functions unnecessarily restart during FunctionRuntimeManager init phase (#5527) No new revisions were added by this update. Summary of changes: .../functions/worker/FunctionRuntimeManager.java | 61 +- .../worker/FunctionRuntimeManagerTest.java | 74 ++ 2 files changed, 80 insertions(+), 55 deletions(-)
[GitHub] [pulsar] wolfstudy merged pull request #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase
wolfstudy merged pull request #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase URL: https://github.com/apache/pulsar/pull/5527 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.6
aahmed-se commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.6 URL: https://github.com/apache/pulsar/pull/5043#issuecomment-548676402 @sijie please take a look at unblock this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet edited a comment on issue #5345: [Doc] Fix doc issues related to BookKeeper, Python Client, and so on
Anonymitaet edited a comment on issue #5345: [Doc] Fix doc issues related to BookKeeper, Python Client, and so on URL: https://github.com/apache/pulsar/issues/5345#issuecomment-548208199 * Inconsistent and incorrect contents have been fixed in https://github.com/apache/pulsar/pull/5528 * Broken links issues have been fixed in https://github.com/apache/pulsar/pull/5496, https://github.com/apache/pulsar/pull/5501, https://github.com/apache/pulsar/pull/5485 * For Python API doc, as discussed with @tuteng, the correct information on http://pulsar.apache.org/api/python/ should be:  This website is automatically generated and cannot be modified from my side. We've reported this issue to ASF, waiting for the response. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet opened a new pull request #5528: [Doc] Add pulsar-admin command and fix inconsistencies
Anonymitaet opened a new pull request #5528: [Doc] Add pulsar-admin command and fix inconsistencies URL: https://github.com/apache/pulsar/pull/5528 Fix https://github.com/apache/pulsar/issues/5345 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet edited a comment on issue #5345: [Doc] Fix doc issues related to BookKeeper, Python Client, and so on
Anonymitaet edited a comment on issue #5345: [Doc] Fix doc issues related to BookKeeper, Python Client, and so on URL: https://github.com/apache/pulsar/issues/5345#issuecomment-548208199 * Inconsistent and incorrect issues have been fixed in https://github.com/apache/pulsar/pull/5528 * Broken links issues have been fixed in https://github.com/apache/pulsar/pull/5496, https://github.com/apache/pulsar/pull/5501, https://github.com/apache/pulsar/pull/5485 * For Python API doc, as discussed with @tuteng, the correct information on http://pulsar.apache.org/api/python/ should be:  This website is automatically generated and cannot be modified from my side. We've reported this issue to ASF, waiting for the response. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet commented on issue #5529: [Doc] Add work flow of checking broken links
Anonymitaet commented on issue #5529: [Doc] Add work flow of checking broken links URL: https://github.com/apache/pulsar/pull/5529#issuecomment-548656840 @tuteng could you please help review? Thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet opened a new pull request #5529: [Doc] Add work flow of checking broken links
Anonymitaet opened a new pull request #5529: [Doc] Add work flow of checking broken links URL: https://github.com/apache/pulsar/pull/5529 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Jennifer88huang commented on issue #5518: python functions cannot find python3 interpreter
Jennifer88huang commented on issue #5518: python functions cannot find python3 interpreter URL: https://github.com/apache/pulsar/issues/5518#issuecomment-548669802 @wolfstudy could you please help look into the issue? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] addisonj commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.6
addisonj commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.6 URL: https://github.com/apache/pulsar/pull/5043#issuecomment-548576779 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] vicaya commented on issue #5417: Broker hangs and crashes when listing non-persistent topics
vicaya commented on issue #5417: Broker hangs and crashes when listing non-persistent topics URL: https://github.com/apache/pulsar/issues/5417#issuecomment-548576865 We're seeing this in 2.4.0 clusters as well. I suspect that bumping the jetty thread count would help. Let's make the jetty thread count configurable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3776: Use at least 8 threads in Jetty thread pool
merlimat commented on issue #3776: Use at least 8 threads in Jetty thread pool URL: https://github.com/apache/pulsar/pull/3776#issuecomment-548596117 @vicaya That is already configurable. The only question was always what was the most conservative default value. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase
jerrypeng commented on issue #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase URL: https://github.com/apache/pulsar/pull/5527#issuecomment-548615501 rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar.wiki] branch master updated: Updated Home (markdown)
This is an automated email from the ASF dual-hosted git repository. rxl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git The following commit(s) were added to refs/heads/master by this push: new aa36301 Updated Home (markdown) aa36301 is described below commit aa363011cc4040b207e5ddcca3e45c13422d7972 Author: 冉小龙 AuthorDate: Fri Nov 1 10:38:28 2019 +0800 Updated Home (markdown) --- Home.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Home.md b/Home.md index 40bd55a..7f67ef2 100644 --- a/Home.md +++ b/Home.md @@ -7,6 +7,7 @@ We encourage to document any big change or feature or any addition to public use ### Proposed +* [[PIP 50: Package Management]] * [[PIP 49: Permission levels and inheritance]] * [[PIP 48: hierarchical admin api]] * [[PIP 47: Time Based Release Plan]]
[GitHub] [pulsar] vicaya commented on issue #3776: Use at least 8 threads in Jetty thread pool
vicaya commented on issue #3776: Use at least 8 threads in Jetty thread pool URL: https://github.com/apache/pulsar/pull/3776#issuecomment-548576143 Can we make the number of jetty threads configurable? Looks like it's impacting a bunch of other issues that involves admin tasks: #5417, #3877, #3878... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #5463: Function runtime pluggable
jerrypeng commented on issue #5463: Function runtime pluggable URL: https://github.com/apache/pulsar/pull/5463#issuecomment-548594823 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] nathan-zhu commented on issue #5514: use pulsar get error in mac os
nathan-zhu commented on issue #5514: use pulsar get error in mac os
URL: https://github.com/apache/pulsar/issues/5514#issuecomment-548624555
i already change file name to others but still get same error.
```
Traceback (most recent call last):
File "testpulsar.py", line 2, in
client = pulsar.Client('pulsar://localhost:6650')
AttributeError: module 'pulsar' has no attribute 'Client'
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] rdhabalia commented on issue #5513: Publish rate limit on broker to avoid OOM
rdhabalia commented on issue #5513: Publish rate limit on broker to avoid OOM URL: https://github.com/apache/pulsar/issues/5513#issuecomment-548624411 @jiazhai have you started seeing OOM recently in 2.5 or it was there before 2.5? are you proposing similar publish rate implemented at #3985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar.wiki] branch master updated: Created PIP 50: Package Management (markdown)
This is an automated email from the ASF dual-hosted git repository. rxl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git The following commit(s) were added to refs/heads/master by this push: new 34de6af Created PIP 50: Package Management (markdown) 34de6af is described below commit 34de6af2027a2d10f5f3963c12a06678050eafa2 Author: 冉小龙 AuthorDate: Fri Nov 1 10:37:40 2019 +0800 Created PIP 50: Package Management (markdown) --- PIP-50:-Package-Management.md | 7 +++ 1 file changed, 7 insertions(+) diff --git a/PIP-50:-Package-Management.md b/PIP-50:-Package-Management.md new file mode 100644 index 000..2b599a3 --- /dev/null +++ b/PIP-50:-Package-Management.md @@ -0,0 +1,7 @@ + + * **Status**: Drafting + * **Author**: Yong Zhang + * **Pull Request**: + * **Mailing List discussion**: + +Design Proposal: https://docs.google.com/document/d/1FDSNhc8YB1yUMFBUrx8p-1nH1s5Kftp04PkFKWec-2U/edit \ No newline at end of file
[GitHub] [pulsar] jerrypeng opened a new pull request #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase
jerrypeng opened a new pull request #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase URL: https://github.com/apache/pulsar/pull/5527 ### Motivation During the init phase of FunctionRuntimeManager, functions may be unnecessarily restarted. This impacts running functions via external schedulers e.g. Kubernetes ### Modifications - Don't set any of the FunctionRuntimeInfo during init time. - Fixed a test to check for check for this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase
merlimat commented on a change in pull request #5527: fix: Functions
unnecessarily restart during FunctionRuntimeManager init phase
URL: https://github.com/apache/pulsar/pull/5527#discussion_r341401565
##
File path:
pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
##
@@ -78,7 +78,37 @@
// All the runtime info related to functions executed by this worker
// Fully Qualified InstanceId - > FunctionRuntimeInfo
@VisibleForTesting
-Map functionRuntimeInfoMap = new
ConcurrentHashMap<>();
+class FunctionRuntimeInfos {
+
+private Map functionRuntimeInfoMap = new
ConcurrentHashMap<>();
+
+public FunctionRuntimeInfo get(String fullyQualifiedInstanceId) {
+return functionRuntimeInfoMap.get(fullyQualifiedInstanceId);
+}
+
+public void put(String fullyQualifiedInstanceId, FunctionRuntimeInfo
functionRuntimeInfo) {
+if (!isInitializePhase) {
+functionRuntimeInfoMap.put(fullyQualifiedInstanceId,
functionRuntimeInfo);
+}
+}
+
+public void remove (String fullyQualifiedInstanceId) {
+if (!isInitializePhase) {
+functionRuntimeInfoMap.remove(fullyQualifiedInstanceId);
+}
+}
+
+public Map getAll() {
+return functionRuntimeInfoMap;
+}
+
+public int size() {
+return functionRuntimeInfoMap.size();
+}
+}
+
+@VisibleForTesting
+FunctionRuntimeInfos functionRuntimeInfos = new FunctionRuntimeInfos();
Review comment:
```suggestion
final FunctionRuntimeInfos functionRuntimeInfos = new
FunctionRuntimeInfos();
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat merged pull request #5526: Python API: add get_input_topics()
merlimat merged pull request #5526: Python API: add get_input_topics() URL: https://github.com/apache/pulsar/pull/5526 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat closed issue #5525: python API missing get_input_topics
merlimat closed issue #5525: python API missing get_input_topics URL: https://github.com/apache/pulsar/issues/5525 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] kevenYLi commented on issue #5516: COMPILATION ERROR:pulsar-kafka-compat-client-test
kevenYLi commented on issue #5516: COMPILATION ERROR:pulsar-kafka-compat-client-test URL: https://github.com/apache/pulsar/issues/5516#issuecomment-548634827 Yes, I did. Steps to build dependencies: - cd pulsar - mvn clean -U install -Dmaven.test.skip=true and the result is success.  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] nathan-zhu edited a comment on issue #5514: use pulsar get error in mac os
nathan-zhu edited a comment on issue #5514: use pulsar get error in mac os
URL: https://github.com/apache/pulsar/issues/5514#issuecomment-548624555
i already change file name to others but still get same error.
```
Traceback (most recent call last):
File "testpulsar.py", line 2, in
client = pulsar.Client('pulsar://localhost:6650')
AttributeError: module 'pulsar' has no attribute 'Client'
```
i try to follow this issue:
https://github.com/apache/pulsar/issues/5248#issuecomment-533939386
i use minicode package, found libpython3.7.dylib at below
```
~/p/P/p/pulsar-client-cpp ❯❯❯ sudo find / -name "libpython3.7.dylib"
✘ 130
Password:
/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7.dylib
/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin/libpython3.7.dylib
```
but get error during cmake:
```
cmake
DPYTHON_LIBRARY="/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7.dylib"
.
```
```
-- BUILD_TESTS: ON
-- BUILD_PYTHON_WRAPPER: ON
-- LINK_STATIC: OFF
-- USE_LOG4CXX: OFF
-- CMAKE_BUILD_TYPE: RelWithDebInfo
-- PYTHON: 2.7.10
CMake Warning at
/usr/local/Cellar/cmake/3.15.4/share/cmake/Modules/FindBoost.cmake:2003
(message):
No header defined for python-mt; skipping header check
Call Stack (most recent call first):
CMakeLists.txt:143 (find_package)
CMake Warning at
/usr/local/Cellar/cmake/3.15.4/share/cmake/Modules/FindBoost.cmake:2003
(message):
No header defined for python-py27; skipping header check
Call Stack (most recent call first):
CMakeLists.txt:143 (find_package)
CMake Warning at
/usr/local/Cellar/cmake/3.15.4/share/cmake/Modules/FindBoost.cmake:2003
(message):
No header defined for python27-mt; skipping header check
Call Stack (most recent call first):
CMakeLists.txt:143 (find_package)
CMake Error at CMakeLists.txt:151 (MESSAGE):
Could not find Boost Python library
-- Configuring incomplete, errors occurred!
See also
"/Users/lidongzhu/python/Pulsar/pulsar-branch-2.4/pulsar-client-cpp/CMakeFiles/CMakeOutput.log".
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #5514: use pulsar get error in mac os
tuteng commented on issue #5514: use pulsar get error in mac os URL: https://github.com/apache/pulsar/issues/5514#issuecomment-548634853 Please send command about how to install pulsar-client? You can try ``` dir(pulsar) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase
wolfstudy commented on issue #5527: fix: Functions unnecessarily restart during FunctionRuntimeManager init phase URL: https://github.com/apache/pulsar/pull/5527#issuecomment-548642617 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5504: [Transaction coordinator]Add transaction metadata store service
codelipenghui commented on issue #5504: [Transaction coordinator]Add transaction metadata store service URL: https://github.com/apache/pulsar/pull/5504#issuecomment-548605023 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (3d5bc0a -> a4c330e)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 3d5bc0a [Test] Migrate deprecated Matchers to ArgumentMatchers (#5423) add a4c330e Python API: add get_input_topics() (#5526) No new revisions were added by this update. Summary of changes: pulsar-client-cpp/python/pulsar/functions/context.py | 5 + pulsar-functions/instance/src/main/python/contextimpl.py | 3 +++ site2/docs/functions-api.md | 1 + site2/docs/functions-develop.md | 2 ++ 4 files changed, 11 insertions(+)
[pulsar] branch master updated (a4c330e -> 79f4710)
This is an automated email from the ASF dual-hosted git repository. rxl pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from a4c330e Python API: add get_input_topics() (#5526) add 79f4710 Fix list non-persistent topics shows the persistent topics (#5502) No new revisions were added by this update. Summary of changes: .../pulsar/broker/admin/v2/NonPersistentTopics.java | 8 +++- .../apache/pulsar/tests/integration/cli/CLITest.java| 17 + 2 files changed, 24 insertions(+), 1 deletion(-)
[GitHub] [pulsar] wolfstudy merged pull request #5502: Fix list non-persistent topics shows the persistent topics
wolfstudy merged pull request #5502: Fix list non-persistent topics shows the persistent topics URL: https://github.com/apache/pulsar/pull/5502 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy closed issue #5414: list non-persistent topics shows the persistent topics
wolfstudy closed issue #5414: list non-persistent topics shows the persistent topics URL: https://github.com/apache/pulsar/issues/5414 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on a change in pull request #5515: [Transaction Coordinator] Bootstrap pulsar system namespace and create TC assign topic.
codelipenghui commented on a change in pull request #5515: [Transaction
Coordinator] Bootstrap pulsar system namespace and create TC assign topic.
URL: https://github.com/apache/pulsar/pull/5515#discussion_r341402476
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
##
@@ -135,6 +144,10 @@ public static void main(String[] args) throws Exception {
arguments.configurationStore = arguments.globalZookeeper;
}
+if (arguments.numTransactionCoordinators <= 0) {
+System.err.println("Number of transaction coordinators must
greater than 0");
Review comment:
yes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #5278: Fixed reader cursor seek with disconnection
merlimat commented on issue #5278: Fixed reader cursor seek with disconnection URL: https://github.com/apache/pulsar/pull/5278#issuecomment-548228194 > If we don't disconnect the consumers, how do consumers reset their state? e.g. pending queue, acknowledgement, and etc. Consumers they won't reset, they just will see the messages restart from an earlier (or later) point in time. I don't believe that should affect the consumer logic since there will be no visible disruption from consumer perspective. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet edited a comment on issue #5345: [Doc] Fix doc issues related to BookKeeper, Python Client, and so on
Anonymitaet edited a comment on issue #5345: [Doc] Fix doc issues related to BookKeeper, Python Client, and so on URL: https://github.com/apache/pulsar/issues/5345#issuecomment-548208199 * Inconsistent and incorrect issues have been fixed in * Broken links issues have been fixed in https://github.com/apache/pulsar/pull/5496, https://github.com/apache/pulsar/pull/5501, https://github.com/apache/pulsar/pull/5485 * For Python API doc, as discussed with @tuteng, the correct information on http://pulsar.apache.org/api/python/ should be:  This website is automatically generated and cannot be modified from my side. We've reported this issue to ASF, waiting for the response. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet edited a comment on issue #5345: [Doc] Fix doc issues related to BookKeeper, Python Client, and so on
Anonymitaet edited a comment on issue #5345: [Doc] Fix doc issues related to BookKeeper, Python Client, and so on URL: https://github.com/apache/pulsar/issues/5345#issuecomment-548208199 * Inconsistent and incorrect issues have been fixed in https://github.com/apache/pulsar/pull/5496, https://github.com/apache/pulsar/pull/5501, https://github.com/apache/pulsar/pull/5485 * Broken links issues have been fixed in * For Python API doc, as discussed with @tuteng, the correct information on http://pulsar.apache.org/api/python/ should be:  This website is automatically generated and cannot be modified from my side. We've reported this issue to ASF, waiting for the response. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #5278: Fixed reader cursor seek with disconnection
merlimat commented on issue #5278: Fixed reader cursor seek with disconnection URL: https://github.com/apache/pulsar/pull/5278#issuecomment-548230834 > If disconnect all consumers while reset cursor, client will create new consumers, that's not going to happen It can still happen, and actually it's more probable to happen the longer the ack is delayed from the receival of the message. Even if the consumer session is recreated, a consumer can receive a message, then seek it's triggered and then it will call ack-cumulative returning the cursor to the previous position. An application that seeks while acking (either individual or cumulative) is bound to unpredictable behavior in any case, irrespective of the disconnection. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch dependabot/maven/com.fasterxml.jackson.core-jackson-databind-2.10.0 updated (3fb4007 -> e4f4136)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.fasterxml.jackson.core-jackson-databind-2.10.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git. discard 3fb4007 Bump jackson-databind from 2.9.9.3 to 2.10.0 add 7abe690 [cpp-build] Do not install libsnappy at each build (#5458) add a9b893f [Doc]Add md suffix for fix cdc link (#5511) add 0076a1b Upgrade dependencies for security fixes (#5232) add e4f4136 Bump jackson-databind from 2.9.9.3 to 2.10.0 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (3fb4007) \ N -- N -- N refs/heads/dependabot/maven/com.fasterxml.jackson.core-jackson-databind-2.10.0 (e4f4136) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: distribution/server/src/assemble/LICENSE.bin.txt | 49 ++--- jclouds-shaded/pom.xml | 11 + pom.xml| 9 ++-- pulsar-client-cpp/docker-build.sh | 2 +- pulsar-client-cpp/docker-lib-check.sh | 23 -- pulsar-client-cpp/docker-tests.sh | 2 +- .../functions/runtime/KubernetesRuntime.java | 5 ++- .../pulsar/functions/runtime/ProcessRuntime.java | 11 ++--- .../pulsar/io/cassandra/CassandraAbstractSink.java | 3 +- .../pulsar/proxy/server/AdminProxyHandler.java | 3 +- pulsar-sql/presto-distribution/LICENSE | 49 +++-- pulsar-sql/presto-distribution/pom.xml | 50 +- pulsar-sql/presto-pulsar-plugin/pom.xml| 1 - .../website/versioned_docs/version-2.3.0/io-cdc.md | 4 +- .../versioned_docs/version-2.4.0/io-quickstart.md | 2 +- tiered-storage/file-system/pom.xml | 11 + 16 files changed, 143 insertions(+), 92 deletions(-) delete mode 100755 pulsar-client-cpp/docker-lib-check.sh
[GitHub] [pulsar] wolfstudy merged pull request #5232: Upgrade dependencies for security fixes
wolfstudy merged pull request #5232: Upgrade dependencies for security fixes URL: https://github.com/apache/pulsar/pull/5232 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (a9b893f -> 0076a1b)
This is an automated email from the ASF dual-hosted git repository. rxl pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from a9b893f [Doc]Add md suffix for fix cdc link (#5511) add 0076a1b Upgrade dependencies for security fixes (#5232) No new revisions were added by this update. Summary of changes: distribution/server/src/assemble/LICENSE.bin.txt | 49 ++--- jclouds-shaded/pom.xml | 11 + pom.xml| 9 ++-- .../functions/runtime/KubernetesRuntime.java | 5 ++- .../pulsar/functions/runtime/ProcessRuntime.java | 11 ++--- .../pulsar/io/cassandra/CassandraAbstractSink.java | 3 +- .../pulsar/proxy/server/AdminProxyHandler.java | 3 +- pulsar-sql/presto-distribution/LICENSE | 49 +++-- pulsar-sql/presto-distribution/pom.xml | 50 +- pulsar-sql/presto-pulsar-plugin/pom.xml| 1 - tiered-storage/file-system/pom.xml | 11 + 11 files changed, 138 insertions(+), 64 deletions(-)
[GitHub] [pulsar] wolfstudy closed issue #4057: Security Vulnerabilities - Black Duck Scan - Pulsar v.2.3.1
wolfstudy closed issue #4057: Security Vulnerabilities - Black Duck Scan - Pulsar v.2.3.1 URL: https://github.com/apache/pulsar/issues/4057 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb opened a new issue #5518: python functions cannot find python3 interpreter
candlerb opened a new issue #5518: python functions cannot find python3
interpreter
URL: https://github.com/apache/pulsar/issues/5518
**Describe the bug**
python functions do not start if the system has "python3" but not "python"
(which is the case for modern Ubuntu systems - "python" is python2 and is not
installed by default)
**To Reproduce**
Write a simple function `func1.py`
```
from pulsar import Function
class FirstFunction(Function):
def process(self, item, context):
log = context.get_logger()
log.info("Got %r with properties %r" % (item,
context.get_message_properties()))
```
Deploy it:
```
apache-pulsar-2.4.1/bin/pulsar-admin functions create --name womble --inputs
my-topic \
--py /home/ubuntu/func1.py --classname FirstFunction --log-topic pulsar-log
```
(side note: have to give absolute path to function, pulsar-admin changes the
working directory)
Check its status:
```
$ apache-pulsar-2.4.1/bin/pulsar-admin functions status --name womble
{
"numInstances" : 1,
"numRunning" : 0,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : false,
"error" : "Cannot run program \"python\": error=2, No such file or
directory",
"numRestarts" : 16,
"numReceived" : 0,
"numSuccessfullyProcessed" : 0,
"numUserExceptions" : 0,
"latestUserExceptions" : [ ],
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"averageLatency" : 0.0,
"lastInvocationTime" : 0,
"workerId" : "c-standalone-fw-ldex-pulsar.int.soundmouse.net-8080"
}
} ]
}
```
Note error: **Cannot run program "python": error=2, No such file or
directory**
**Expected behavior**
I would expect the function to deploy.
**Screenshots**
N/A
**Desktop (please complete the following information):**
N/A
**Additional context**
This may be the same problem as #5369 - however in that case, it's the
shebang line in pulsar's own python code, whereas this one is my own function
which is not starting.
I cannot find it stated in Pulsar documentation whether Pulsar supports
python2, python3 or both. However python2 is [very
shortly](https://pythonclock.org/) going to be end-of-life.
**Workarounds**
Install python2 and write my code in python2. I really don't want to do
that, with python2 being end-of-life in 2 months' time.
Symlink python to python3. However in Ubuntu systems, symlinking python to
python3 has the potential to cause all sorts of problems. Many applications and
scripts depend on `python` meaning `python2` (in which case they will depend on
the `python-XXX` packages rather than `python3-XXX`). More content
[here](https://askubuntu.com/questions/320996/how-to-make-python-program-command-execute-python-3#answer-475815).
Ideally pulsar would try python3 and fallback to python. If pulsar were to
be hard-coded to run "python3" instead of "python" that would also be fine IMO,
given the impending EOL of python2.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] KannarFr opened a new issue #5519: Zookeeper issue
KannarFr opened a new issue #5519: Zookeeper issue URL: https://github.com/apache/pulsar/issues/5519 **Describe the bug** Hi, Today, one of our zookeeper node had the following error: ``` Oct 31 10:01:06 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:06.697 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e569 zxid:0x450025f5be txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Oct 31 10:01:13 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:13.316 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e575 zxid:0x450025f5cb txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Oct 31 10:01:13 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:13.523 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x3037b7082ea00fa type:delete cxid:0x6d588 zxid:0x450025f5ce txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Oct 31 10:01:35 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:35.465 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e58a zxid:0x450025f5d6 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Oct 31 10:01:49 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:49.277 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x3037b7082ea00fa type:delete cxid:0x6d5ad zxid:0x450025f5e1 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Oct 31 10:02:30 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:02:30.556 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e5b5 zxid:0x450025f5ea txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Oct 31 10:04:08 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:04:08.683 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x3037b7082ea00fa type:delete cxid:0x6d605 zxid:0x450025f5f5 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 ``` Then no log and zk node seems stucked. We restarted it and after a while (10 minutes), it recovered alone. WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb commented on issue #5518: python functions cannot find python3 interpreter
candlerb commented on issue #5518: python functions cannot find python3 interpreter URL: https://github.com/apache/pulsar/issues/5518#issuecomment-548329211 Hmm, I found [docker/pulsar/scripts/set_python_version.sh](https://github.com/apache/pulsar/blob/master/docker/pulsar/scripts/set_python_version.sh) which suggests that people *are* making python a symlink to python3 ``` sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10 ``` As I say, this is generally frowned upon. I found there is pythonInstanceLocation in WorkerConfig.java. The documentation says "change the python instance location only when you put the python instance jar in a different location" - but I'm looking for the python interpreter, not a jar file. (I don't *think* it's using anything I haven't yet found the point where pulsar execs the python interpreter. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] candlerb commented on issue #5369: "#!/usr/bin/env python" doesn't work on Ubuntu 18.04 / python3
candlerb commented on issue #5369: "#!/usr/bin/env python" doesn't work on Ubuntu 18.04 / python3 URL: https://github.com/apache/pulsar/issues/5369#issuecomment-548330379 Actually, I believe the shebang lines in these python source files are spurious, since they do not have the eXecute bit set anyway: ``` $ tar -tvzf apache-pulsar-2.4.1-bin.tar.gz | grep python_instance -rw-r--r-- jia/staff 10165 2019-08-28 08:59 apache-pulsar-2.4.1/instances/python-instance/python_instance_main.py -rw-r--r-- jia/staff 3384 2019-08-28 08:59 apache-pulsar-2.4.1/instances/python-instance/tests/test_python_instance.py -rw-r--r-- jia/staff 20104 2019-08-28 08:59 apache-pulsar-2.4.1/instances/python-instance/python_instance.py ``` Shebangs are only needed if you `exec()` the file directly. So I'm guessing pulsar is explicitly running the interpreter, like `/usr/bin/python .py`, but I can't find that code right now. If it's true, then I think it would be good idea to remove all the shebang lines to avoid confusion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai opened a new issue #5513: Publish rate limit on broker to avoid OOM
jiazhai opened a new issue #5513: Publish rate limit on broker to avoid OOM URL: https://github.com/apache/pulsar/issues/5513 Out Of Memory is seen frequently while client produced message too quickly and used all the direct memory. This happens like this: 1. Client send message to broker; 2. Broker call bookkeeper client to write message to Bookie, and hold the direct memory until write success; If in the 2nd step the direct memory not get released, then the Broker will get OOM. It would be great to have a way to detect the memory pressure, and do the publish rate limit if lack of memory; once the memory get back, cancel the rate limit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide
Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide URL: https://github.com/apache/pulsar/issues/5015#issuecomment-524149120 I've created the following PRs to improve the doc overall: # Overview * https://github.com/apache/pulsar/pull/5183 * https://github.com/apache/pulsar/pull/5224 # Use * https://github.com/apache/pulsar/pull/5071 # Built-in connector Overall: https://github.com/apache/pulsar/issues/5015 Connectors come into two types: ## Source connector Source connector name|Configuration|Example |---|---|--- Kafka |https://github.com/apache/pulsar/pull/5203|https://github.com/apache/pulsar/pull/5203| * https://github.com/apache/pulsar/pull/5168 * https://github.com/apache/pulsar/pull/5173 * https://github.com/apache/pulsar/pull/5197 * https://github.com/apache/pulsar/pull/5199 * https://github.com/apache/pulsar/pull/5210 * https://github.com/apache/pulsar/pull/5211 * https://github.com/apache/pulsar/pull/5214 * https://github.com/apache/pulsar/pull/5226 ## Sink connector Sink connector name|Configuration|Example |---|---|--- Flume|https://github.com/apache/pulsar/pull/5238|| JDBC|https://github.com/apache/pulsar/pull/5275|| MongoDB|https://github.com/apache/pulsar/pull/5289|| RabbitMQ|https://github.com/apache/pulsar/pull/5290|| Redis|https://github.com/apache/pulsar/pull/5291|| Solr|https://github.com/apache/pulsar/pull/5292|| * Flume sink: * HDFS3 sink: https://github.com/apache/pulsar/pull/5239 * File source + **example**: https://github.com/apache/pulsar/pull/5240 * Twitter Firehose source: https://github.com/apache/pulsar/pull/5251 * Netty source: https://github.com/apache/pulsar/pull/5253 * Rabbit source: https://github.com/apache/pulsar/pull/5255 * Aerospike sink: https://github.com/apache/pulsar/pull/5256 * Cassandra sink: https://github.com/apache/pulsar/pull/5257 * ElasticSearch sink: https://github.com/apache/pulsar/pull/5258 * InfluxDB sink: https://github.com/apache/pulsar/pull/5273 # CDC connector * https://github.com/apache/pulsar/pull/5198 # Debug * https://github.com/apache/pulsar/pull/5041 # Connector Admin CLI * https://github.com/apache/pulsar/pull/5028 * https://github.com/apache/pulsar/pull/5184 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat merged pull request #5488: Python API: Add test case for create_reader(is_read_compacted)
merlimat merged pull request #5488: Python API: Add test case for create_reader(is_read_compacted) URL: https://github.com/apache/pulsar/pull/5488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (0076a1b -> d9ebec5)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 0076a1b Upgrade dependencies for security fixes (#5232) add d9ebec5 Python API: Add test case for create_reader(is_read_compacted) (#5488) No new revisions were added by this update. Summary of changes: pulsar-client-cpp/python/pulsar_test.py | 69 - 1 file changed, 42 insertions(+), 27 deletions(-)
[GitHub] [pulsar] codelipenghui commented on issue #5504: [Transaction coordinator]Add transaction metadata store service
codelipenghui commented on issue #5504: [Transaction coordinator]Add transaction metadata store service URL: https://github.com/apache/pulsar/pull/5504#issuecomment-548256197 run java8 tests run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
