kyrozetera opened a new issue #9089: URL: https://github.com/apache/pulsar/issues/9089
**Describe the bug** Pulsar is not respecting the `isSchemaValidationEnforced=false` or `isAllowAutoUpdateSchema=true` in the broker.conf on startup. Though validation appears to be disabled when checking the value through the API, it still does the validation and fails on conflicting schemas if the schema has been updated. When manually disabling the validation through the API, then it begins to function as expected. To get `isAllowAutoUpdateSchema` to function properly, it was a bit more involved as I had to disable and enable both it and the schema validation through the API before it started working. I'm using the debezium connector with postgres, for pulsar 2.7.0 and this is where I'm seeing this behavior (same behavior in 2.6.1 & 2.6.2). **To Reproduce** Steps to reproduce the behavior: 1. Setup Debezium source connector following by https://pulsar.apache.org/docs/en/io-cdc-debezium/ 2. Ensure config is set `isSchemaValidationEnforced=false` in broker.conf or standalone.conf 3. Start pulsar standalone `bin/pulsar standalone` 4. Start debezium connector `bin/pulsar-admin source localrun --source-config-file conf/debezium-postgres-source-config.yaml` 5. Check schema validation: `bin/pulsar-admin namespaces get-schema-validation-enforce public/default` - `false` 6. Create table and insert record: ```sql CREATE TABLE public.test_table ( id int4 NOT NULL GENERATED BY DEFAULT AS IDENTITY, campaign_id int4 NOT NULL, "name" varchar NULL, start_date timestamp NOT NULL, end_date timestamp NOT NULL, CONSTRAINT test_table_campaign_id_name_unique UNIQUE (campaign_id, name), CONSTRAINT pk_test_table_id PRIMARY KEY (id) ); INSERT INTO test_table (campaign_id, name, start_date, end_date) VALUES (1, 'test', '2020-12-01', '2020-12-31'); ``` 7. Alter column, then insert again: ```sql ALTER TABLE test_table ALTER COLUMN end_date DROP NOT NULL; INSERT INTO test_table (campaign_id, name, start_date, end_date) VALUES (2, 'test', '2020-12-01', '2020-12-31'); ``` 8. See error: ``` 00:06:02.592 [pulsar-client-io-1-2] WARN org.apache.pulsar.client.impl.ProducerImpl - [public/default/koddi-console-1.public.test_table] [standalone-0-2] GetOrCreateSchema error org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: org.apache.avro.SchemaValidationException: Unable to read schema: ``` 9. Disable via API: `bin/pulsar-admin namespaces set-schema-validation-enforce public/default --disable` 10. Restart debezium connector 11. Insert again and see no validation error in debezium/pulsar ```sql insert into test_table (campaign_id, name, start_date, end_date) VALUES (3, 'test', '2020-12-01', '2020-12-31'); ``` **Expected behavior** The schema should not be validated, or it should auto update when enabled **Screenshots** N/A **Desktop (please complete the following information):** - OS: Linux (Ubuntu 20.04 / Fedora / Debian10) **Additional context** Having the same issue with a standalone instance locally and a pulsar cluster ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
