GitHub user rob-kvietkauskas-ign added a comment to the discussion: Functions
worker configuration for bare-metal cluster
Pulsar instance is deployed same directory on (each broker machine) to
`/opt/pulsar/apache-pulsar-<version>`;
Connectors (NAR files) are respectively stored in
`/opt/pulsar/apache-pulsar-<version>/connectors`;
Connector configurations (YAML files) are respectively stored in
`/opt/pulsar/apache-pulsar-<version>/connectors/configuration`;
Until last thursday, every time I tried to create (register) a connector
(independent on type of it), I was performing following (working directory –
`/opt/pulsar/apache-pulsar-<version>/bin`) for `source` – same way it was
described in examples:
```
./pulsar-admin sources create \
--archive
/opt/pulsar/apache-pulsar-2.9.1/connectors/pulsar-io-debezium-postgres-2.9.1.nar
\
--name postgres-test \
--destination-topic-name source-connector-topic \
--source-config-file
/opt/pulsar/apache-pulsar-2.9.1/connectors/configuration/debezium-postgres-source-config.yaml
```
and following for `sink` connectors:
```
./pulsar-admin sinks create \
--sink-config-file
/opt/pulsar/apache-pulsar-2.9.1/connectors/configuration/sink-connector.yaml \
--name "sink-connector" \
--archive
/opt/pulsar/apache-pulsar-2.9.1/connectors/custom-cdc-sink-connector.nar \
--parallelism 3 \
--inputs
persistent://public/default/db_name.schema_name.table_observed_by_source_name
```
In bare-metal cluster such calls ended up with internal pulsar errors (related
to ledger allocation failures). `localrun` calls with same parameters in same
bare-metal clusters were successful. Only `create` was failing. On thursday,
after reading same `pulsar-admin sources create` documentation line () about
`--archive` parameter for a thousand time, I diceded to add `file://` schema
prefix to previously supplied `--archive` parameter value and that finally
ended up in successfull `create` call (connector got registered successfully
and was started automatically afterwards). The line was:
```
-a, --archive
The path to the NAR archive for the Source. It also supports url-path
[http/https/file (file protocol assumes that file already exists on
worker host)] from which worker can download the package.
```
I performed some more testing both in bare-metal cluster running 2.11.1 and
2.9.1. Even though I was able to `create` connectors without `file://` schema
in `2.11.1` cluster with `functionsWorkerEnablePackageManagement` enabled,
connectors didn't start successfully after restart of the broker nodes – they
ended up with internal errors. After deleting connectors and then recreating
them while using `file://` schema prefix, (same) connectors (with same
configure) were recovering successfully after restart.
As I have mentioned before – I wasn't able to successfuly enable
`functionsWorkerEnablePackageManagement` on `2.9.1` bare-metal cluster, so
latest actions in `2.9.1` bare-metal cluster were performed with
`functionsWorkerEnablePackageManagement` disabled.
GitHub link:
https://github.com/apache/pulsar/discussions/20796#discussioncomment-6526947
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
