2019-07-08 09:22:51 UTC - Mahesh: Hi, I have a question regarding pulsar
functions.
1) If the input topic is regex pattern, in what subscription mode does the
function run ?
2) Can auto creation of topics be disabled in pulsar ?
----
2019-07-08 11:28:43 UTC - Yuji Kawamoto: @Yuji Kawamoto has joined the channel
----
2019-07-08 12:00:05 UTC - Ganga Lakshmanasamy: @Karthik Ramasamy I would like
to know if there is any prebuilt code available in pulsar to ingest data from
hadoop to elasticsearch after running the data over a business rule engine like
drools. If you already have any integrations done please share the details.
This is for a requirement from a leading bank where they want the hadoop logs
to be pushed to Elasticsearch. @Ravi
----
2019-07-08 13:15:49 UTC - Ganga Lakshmanasamy: @Ravi Please check.
----
2019-07-08 15:42:15 UTC - David Kjerrumgaard: @Ganga Lakshmanasamy There is a
Sink Connector for ElasticSearch that I wrote, but you would have to write the
other two components. FWIW, there is an File Source connector that I wrote
which you could use as a guide for writing the HDFS source.
----
2019-07-08 16:10:19 UTC - Chris DiGiovanni: I was looking at the instructions
on connecting Presto to Pulsar. After looking at Presto configuration options
of the plugin and the docs, it looks like it doesn't support Authentication and
Authorization. Hoping that the options were just not documented I looked at
the source and it does not look like it supports passing options for TLS,
Athenz, or Kerberos. Is this on the roadmap to support in a future release? I
also cannot find reference in the source or docs but does this plugin handle
reading from offloaded ledgers in S3 as well?
----
2019-07-08 16:11:32 UTC - Ganga Lakshmanasamy: ok. Can you please share the
link for Sink connector.
----
2019-07-08 16:12:47 UTC - David Kjerrumgaard:
<https://github.com/apache/pulsar/blob/master/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchSink.java>
----
2019-07-08 16:13:07 UTC - Ganga Lakshmanasamy: Thanks David
----
2019-07-08 16:19:18 UTC - David Kjerrumgaard: No problem.. If you want a link
to the File source let me know..... :smiley:
----
2019-07-08 16:19:39 UTC - Ganga Lakshmanasamy: sure:+1:
----
2019-07-08 16:25:03 UTC - David Kjerrumgaard:
<https://github.com/apache/pulsar/tree/master/pulsar-io/file/src/main/java/org/apache/pulsar/io/file>
----
2019-07-08 16:31:30 UTC - Addison Higham: oh, 2.4.0 release didn't get
announced here? congrats! :party-parrot:
beers : Matteo Merli, David Kjerrumgaard, Sijie Guo, Ali Ahmed
----
2019-07-08 16:57:16 UTC - Ravi: Thanks ganga, @Karthik Ramasamy this is a demo
for a large bank, we can talk about it when you get a moment. They would like
to see how giggso can manage their big data environment.
----
2019-07-08 17:53:17 UTC - Sijie Guo: @Sijie Guo set the channel topic: Pulsar
graduated as a Top-Level project :tada: - Pulsar release 2.4.0 -
<http://pulsar.apache.org/release-notes/#2.4.0>
heart : Ravi, Ali Ahmed
tada : Karthik Ramasamy, Ali Ahmed, Devin G. Bost
----
2019-07-08 17:53:31 UTC - Sijie Guo: Update the channel description
:slightly_smiling_face:
----
2019-07-08 20:51:12 UTC - Devin G. Bost: @Jerry Peng When I attempt to run
PulsarFunctionE2ETest from the Pulsar source code when I'm on the VPN, I get
this error:
```
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException:
java.lang.RuntimeException: Can't create a producer on assignment topic
<persistent://external-repl-prop/use/pulsar-function-admin/assignment>
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:448)
at
org.apache.pulsar.io.PulsarFunctionE2ETest.setup(PulsarFunctionE2ETest.java:189)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at
org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
at
org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:451)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:516)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:707)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:979)
at
org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1187)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1116)
at org.testng.TestNG.runSuites(TestNG.java:1028)
at org.testng.TestNG.run(TestNG.java:996)
at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Can't create
a producer on assignment topic
<persistent://external-repl-prop/use/pulsar-function-admin/assignment>
at
org.apache.pulsar.functions.worker.WorkerService.start(WorkerService.java:204)
at
org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1044)
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:441)
... 28 more
Caused by: java.lang.RuntimeException: Can't create a producer on assignment
topic <persistent://external-repl-prop/use/pulsar-function-admin/assignment>
at
org.apache.pulsar.functions.worker.SchedulerManager.createProducer(SchedulerManager.java:136)
at
org.apache.pulsar.functions.worker.SchedulerManager.<init>(SchedulerManager.java:94)
at
org.apache.pulsar.functions.worker.WorkerService.start(WorkerService.java:152)
... 30 more
```
I'm going to try running it when off the VPN.
----
2019-07-08 20:51:30 UTC - Devin G. Bost: That's on the 2.3.2 release of the
source code.
----
2019-07-08 20:52:12 UTC - Devin G. Bost: I can update to 2.4.0 if any changes
might affect that test.
----
2019-07-08 20:54:21 UTC - Devin G. Bost: Here's what I got when I ran the tests
when not on the VPN:
----
2019-07-08 20:54:24 UTC - Devin G. Bost:
----
2019-07-08 21:36:52 UTC - Jerry Peng: @Devin G. Bost so most of them pass
----
2019-07-08 21:37:20 UTC - Devin G. Bost: Yes. I'm investigating what's
different compared to my integration test method.
----
2019-07-08 21:37:56 UTC - Jerry Peng: there maybe race conditions in the tests
as we wait only for limited amount of time for operations to complete. Try
running the failed tests a couple times to see if get errors consistently
----
2019-07-08 21:38:23 UTC - Jerry Peng: but my guess is that while you are on the
VPN, you local networking settings are changed
----
2019-07-08 21:39:17 UTC - Jerry Peng: so that localhost doesnt resolve to
127.0.0.1
----
2019-07-08 22:19:44 UTC - Devin G. Bost: @Jerry Peng Thanks for the feedback.
I'll check on that in a bit. I copied the vanilla E2E function into my project,
and it ran successfully. However, after I changed the admin user "superUser" to
"superuser" to match the environment for my other integration tests, the
vanilla E2E function fails with this message:
```
16:18:20.626 [pulsar-io-62-1] INFO o.a.pulsar.broker.service.ServerCnx -
[/127.0.0.1:56910] Client successfully authenticated with tls role superUser
and originalPrincipal null
16:18:20.669 [pulsar-io-62-1] WARN o.a.p.broker.web.PulsarWebResource -
[<persistent://osp/use/pulsar-function-admin/assignment>] Role superUser is not
allowed to lookup topic
16:18:20.680 [pulsar-io-62-1] WARN o.a.p.b.a.impl.PersistentTopicsBase -
Failed to authorize superUser on cluster
<persistent://osp/use/pulsar-function-admin/assignment>
16:18:20.682 [pulsar-io-62-1] WARN o.a.pulsar.broker.service.ServerCnx -
Failed to authorize superUser at [/127.0.0.1:56910] on topic
<persistent://osp/use/pulsar-function-admin/assignment> : Authorization failed
superUser on topic <persistent://osp/use/pulsar-function-admin/assignment> with
error Don't have permission to administrate resources on this tenant
```
----
2019-07-08 22:20:44 UTC - Devin G. Bost: Where else could the `superUser` role
be coming from? Is there a token file or something?
----
2019-07-08 23:35:08 UTC - Jerry Peng: @Devin G. Bost its set in the test
----
2019-07-08 23:36:04 UTC - Jerry Peng: and the certs
----
2019-07-08 23:36:06 UTC - Devin G. Bost: I changed every case to `superuser`,
but it was still referencing `superUser`
----
2019-07-08 23:36:11 UTC - Devin G. Bost: Ah, it must be coming from the certs.
----
2019-07-08 23:36:29 UTC - Jerry Peng: The certs its using is assigned to
superUser
----
2019-07-08 23:36:35 UTC - Devin G. Bost: Got it. Thanks.
----
2019-07-09 01:05:41 UTC - Jim Lambert: I’ve gotten a bit closer with mTLS:
<https://github.com/Jim-Lambert-Bose/test-standalone-pulsar-with-tls>
I’m still seeing function worker TLS errors even though I’ve tried to turn off
the function workers with: `functionsWorkerEnabled=false` in `standalone.conf`.
The good news: it appears to try 5 attempts at function worker connections and
then gives up. So, I’m moving to my next issue.
```
$ pulsar-admin tenants list
HTTP 401 Unauthorized
Reason: HTTP 401 Unauthorized
```
standalone.conf: `superUserRoles=standalone-pulsar-admin`
my cert info:
```
openssl x509 -in /pulsar/client-cert.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
0c:3f:3a:7c:93:25:5c:a6:a0:ab:a8:af:90:9a:5c:77
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = mkcert development CA, OU = root@5d279c58de08, CN = mkcert
root@5d279c58de08
Validity
Not Before: Jun 1 00:00:00 2019 GMT
Not After : Jul 9 00:33:04 2029 GMT
Subject: O = mkcert development certificate, OU = root@37b254048010
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
keyid:B7:35:1C:D7:2B:43:44:29:1E:24:28:66:37:79:DA:A2:1E:EA:5B:D8
X509v3 Subject Alternative Name:
DNS:standalone-pulsar-admin
```
This all looks good to me.
The server log for the request:
```
01:03:28.357 [pulsar-web-55-6] WARN
org.apache.pulsar.broker.web.AuthenticationFilter - [127.0.0.1] Failed to
authenticate HTTP request: Authentication required
01:03:28.383 [pulsar-web-55-6] INFO org.eclipse.jetty.server.RequestLog -
127.0.0.1 - - [09/Jul/2019:01:03:28 +0000] "GET /admin/v2/tenants HTTP/1.1" 401
0 "-" "Pulsar-Java-v2.3.1" 104
```
----
2019-07-09 01:26:54 UTC - Jim Lambert: Hum… I’m beginning to think that Pulsar
doesn’t properly handle certs with `X509v3 Subject Alternative Name`
----
2019-07-09 01:27:25 UTC - Jim Lambert: Could pulsar be searching for the `CN`
of the client cert `Subject`?
----
2019-07-09 04:32:43 UTC - vikash: @Sijie Guo is Key_Shared subscription is
supported on 2.4.0?
----
2019-07-09 04:33:23 UTC - Sijie Guo: yes
----
2019-07-09 05:30:38 UTC - vikash: backlog is showing so high ,how to
clean or schedule for clean
----
2019-07-09 05:46:40 UTC - Penghui Li: Using pulsar admin to clear backlog:
```
bin/pulsar-admin topics clear-backlog -s <sub_name> <topic_name>
```
----
2019-07-09 05:48:58 UTC - Penghui Li: Or rest API:
<http://pulsar.apache.org/admin-rest-api/#operation/skipAllMessages>
----
2019-07-09 05:57:04 UTC - vikash: thanks ,it worked
----
2019-07-09 06:17:44 UTC - Dennis Sehalic: Is there an instruction somewhere
about upgrade best practice? I can't seem to find anything in the docs
----
2019-07-09 07:08:15 UTC - Dennis Sehalic: found this, not sure if it's up to
date though
<https://github.com/apache/pulsar/blob/a8a1453486de6e94a76284503f80c3148bf819c3/site/docs/latest/admin/Upgrade.md>
----
2019-07-09 07:45:58 UTC - Sijie Guo: The general process is unchanged. in
general pulsar bookkeeper keep backward compatibility very well.
----
2019-07-09 08:16:23 UTC - Dennis Sehalic: thanks mate
should I create a git issue on updating the docs site with upgrade instruction
and best practices?
----
2019-07-09 08:25:20 UTC - Sijie Guo: the one you found is actually from my pull
request and the pull request was out there
----
2019-07-09 08:35:34 UTC - Dennis Sehalic: ok
----
